Mekong IP & MSC Workshop Report

CPWF-BFP-Mekong Impact Pathway and Most Significant Change Workshop Report

13th to 15th February, 2006

MekongRiver Commission, Vientiane, Laos

Boru Douthwaite[1] and Sanjini de Silva[2]

Introduction

On 1st October 2005 Phase 1 of the CPWF Impact Assessment Project (IA Project) began. It is a part of the Basin Focal Project (BFP) Initiative and will work in the Volta, Mekong and Karkheh basins. Phase 2 will work in the remaining basins. The project is lead by the InternationalCenter for Tropical Agriculture (CIAT – Spanish Acronym).

The IA project focuses on carrying out ex-ante impact assessment on work carried out by the CPWF projects in the basins. The basin-specific focal projects (Volta, Mekong, Karkheh and São Francisco) focus on more fundamental questions about the extent to which water can influence livelihoods and poverty, and as such are carrying out a different sort of ex-ante impact assessment.

The rationale of the IA Project is captured in Figure 1. In summary, the CPWF (Challenge Program for Water and Food) needs a better appreciation of the existing and potential impact of research on water use in agriculture to justify current and future funding. At the same time CPWF projects would benefit from a better understanding of how and what impact they hope to attain, and a monitoring and evaluation approach that both fosters and tracks progress towards achieving impact. The IA project aims to contribute to both of these requirements.

The IA project has three components:

1)Construction of impact pathways of CPWF projects and piloting of the Most Significant Change (MSC) approach to monitor project progress along its impact pathway;

2)Analysis of the potential extrapolation domain (potential for scaling out and scaling up) of selected project outputs;

3)Scenario analysis that extrapolates the impact of selected high-potential research outputs to the global level.

Figure 1[B1]: I still think this figure one is no good as a stand alone without explanations regarding the 5 why method, what a determinant is, how come you read from 3-1, etc. Very unclear. (3rd why and examples of these approaches (which?) is the first things you read?

The workshop deliverables were are:

1)Draft impact pathways produced for each project

2)Workshop participants able to construct project impact pathways and design a MSC procedure to monitor progress along an impact pathway.

A project impact pathway is: (i)the causal chain of events and outcomes that link outputs to the goal; and (ii) A network map that shows (or plural?) the relationships between project implementing organizations, boundary partners and beneficiaries that are necessary to achieve the goal. After the workshop, the IA project will work with the individual projects to develop project impact narratives. A project impact narrative is a document that describes the project’s rationale. It describes the outputs, outcomes, assumptions, links and relationships shown in the project impact pathway. It weaves together the chain of outcomes with the evolution of the partner relationships (shown in the network maps developed as part of the impact pathway). It is quantified and substantiated by literature and expert opinion as far as possible.

Most Significant Change (MSC) involves the periodic collection of significant change stories resulting from a project's implementation (following its impact pathway) and the subsequent selection of the most significant changes. The regular discussion and selection of change stories fosters an on-going re-evaluation of what really contributes to impact in a project. The changes identified as significant beyond a local context are circulated within the project or program, thus stimulating cross-fertilization of ideas and innovation ().

The Mekong BasinWorkshop

The workshop was funded by the IA Project. Sanjini de Silvia from the IMWI-CPWF Knowledge Sharing in Research Project co-facilitated.

Impact Pathways

In preparation for the workshop, we sent each project a problem tree derived by us from their respective project documents. One or two participants attended from each project (see Participants List, Annex 1). Observers (4) joined groups that interested them.

The structure of the workshop followed the Road Map shown in Figure 2. Participants began by modifying their problem trees (sent to them before the workshop) and then presented them in plenary (Figure 3).

Figure 2:

Figure 3: Example of a project problem tree[B2] box #4, 4th why is incomplete. The arrows are kind of confusing in this one- change for another example?

Participants then listed their project objectives followed by a visioning exercise based on the following:

“You wake up two years after the end of your project. Your project has been a success and is well on its way to achieving its goal. Describe what this success looks like:

  • What is happening differently now?
  • Who is doing what differently?
  • What have been the changes in the lives of the people using the project outputs, and who they interact with?
  • How are project outputs scaling out and scaling up?”

Figure 5: Example of a project vision

[B3]

After presenting the project visions in plenary, the participants then went on to develop a timeline of events and outcomes that explains what has to happen to achieve their visions.

The project outputs, vision and timeline are one part of an impact pathway. The second part is network maps that show the evolution of partnerships necessary for a project to move along its timeline, and achieve its goal (vision).

We asked participants to construct two network maps, one for the present and one for 2 years after the end of their respective project. It is this latter network , that will be achieving the respective project’s impact through scaling-out and scaling-up of project outcomes. Scaling-out is the spread of a project outcomes (i.e., changes such as the use of a new technology, a new strategy, etc.) from farmer to farmer, community to community, within the same stakeholder groups. Scaling-up is an institutional expansion, based largely on first-hand experience, word-of-mouth and positive feed back, from adopters and their grassroots organizations to policy makers, donors, development institutions, and the other stakeholders key to building a more enabling environment for scaling-out process. In other words, scaling-up is the process by which policies, norms, mental models, etc., change in such a way as to support a scaling-out (adoption) process.

We asked participants to identify the ultimate beneficiaries and prioritize the links to them as a way of identifying their project’s key scaling out pathways. In the same way we asked them to identify the stakeholders that could most affect the environment for the spread of their project outputs, and prioritize those links, as a way of identifying their project’s key scaling up pathways.

Figure 6: Example of “now” and “future” network maps

We also asked participants to fully describe their network maps as matrices. We will enter this information into UNICETUCINET, a program for drawing and analyzing networks. This will allow further analysis during the write up of the impact narratives.

Most Significant Change

The third day of the workshop was dedicated to an introduction to MSC and a discussion of its potential uses in the CPWF. We carried out a MSC cycle which included:

1)Selecting a domain for writing significant change stories;

2)Each project writing a story

3)Identification of selection criteria

4)Selection of winning story by selection committee

Participants chose to write significant change stories about project impact (not necessarily their CPWF project) in the last year. The winning story was actually fictitious, illustrating the importance of verifying significant change stories!! After the process, participants reflected on the selection process, and made the following comments:

  • Shows the importance of providing detail in significant change stories, focusing on one change and providing details about human elements.
  • Shows the danger of making up farmer interviews, because these are easily spotted and not credible
  • None of the stories written were about failures
  • The stories often lacked a causal explanation of how project outputs led to outcomes.

Participants were asked about their impression of MSC. Their replies included:

  • Useful and interesting
  • Need more clarity on target group for the stories
  • Is it a tool for learning or for marketing?
  • Doable
  • Useful way of summarizing past events
  • A very good communication tool
  • A good way of collecting details from a large number of projects
  • Selection criteria influenced by power

The comments reflected a concern about the possibility of the selection process being influenced by the power of people in the selection committee, and the desire to select “success” stories for marketing purposes, rather than prioritizing the potential for leaning.

We then asked the participants to suggest possible uses of MSC in the CPWF-Mekong projects. These were:

  • For the monitoring and evaluation of livelihoods
  • To enhance knowledge sharing in farmer group meetings
  • To monitor changes in perspectives of project partners
  • To measure impact of participatory workshops in Companion Modeling Project in Bhutan (actual use)

The Companion Modeling project pointed out that they are already using MSC in their project in Bhutan.

Workshop Evaluation and Feedback

We asked participants what they will use from the impact pathway workshop. Their answers included:

  • Will use impact pathways in future design of projects (x2)
  • Will seek more collaboration with other CPWF projects
  • Will share the approach with other colleagues and partners (x2)
  • Will use the approach to further develop my project
  • Will use the visioning exercise and the drawing of network maps

We also asked people to write on cards what they liked about the workshop after Day 1 and Day 2, and what they suggest we change next time. The results are listed below.

What was good about the workshop

Day 1 evaluation

  • Useful explanation and distinction between outputs and outcomes
  • Preparation of the problem trees before the workshop
  • Working in teams
  • I now have more clarity on the concepts introduced
  • The approach presented is useful for planning

Day 2 evaluation

  • The opportunity to step back to have a broader perspective of our CP project and others
  • Learned a lot in the workshop to share afterwards
  • Excellent note taking on flip charts
  • The workshop facilitators were excellent. They were very experienced and professional
  • Useful tool for project team
  • I got an idea of the diversity of CPWF projects
  • Good guidance between guidance and autonomy of thegroups[B4]
  • The dynamics of networks is useful to envision the future
  • The network matrix after 2 years
  • The guidance on how to write the vision statement
  • Well organized (x2)
  • Work in groups
  • Good facilitation
  • The network matrices
  • The workshop presented good methodology to improve impact
  • Good opportunity for exchange with other projects
  • Facilitators very attentative to the progress of our work
  • The documents prepared about our projects before and after the workshop
  • Good techniques for project presentation and evaluation

To change for next time

Day 1 evaluation
  • Improve time management
  • Group presentation of timelines took a bit long
  • Facilitator should take more time to comment on problem trees
  • A clearer explanation of the why system
  • More information about the other projects
Day 2 evaluation
  • Provide more examples to illustrate points
  • Provide more feedback on exercises
  • Incorporation of impact pathways more formally into the CPWF
  • Better time management to have a shorter workshop
  • More time for group work
  • Better schedule with realistic time management
  • To receive the workshop agenda before the workshop
  • Provide a map of the CPWF project network in the Mekong
  • Reduce the number of projects in the workshop. Nine is too many. Three or four similar projects is optimal
  • Take an example of a completed impact pathway to concretely demonstrate the steps to be taken and documents to be produced
  • Handouts of a completed example would have saved time
  • Examples to illustrate points
  • The groupings of projects were not effective
  • The objective of sharing information between projects was achieved, but there was not much willingness to go further because I think the projects are too different
  • Clearer suggestions for participation preparation in advance
  • Participants sometimes focus on their work and fail to listen to the speaker
  • Some methodological issues need further attention (e.g., network dynamics; definition of terms)

Next Steps

Impact pathways and narratives: The IA project will process the workshop materials (photos were taken of all the outputs corresponding to the boxes in Figure 2) and write the first draft of an impact narrative for each project. We will then work with the respective projects to produce a final product or products, taking into account both the needs of the individual projects and the CP Secretariat.

Most Significant Change: Interested projects will communicate their interest to the IA and KS projects and we will then plan to carry out pilots. It is also possible the CP Secretariat will wish to ask projects to provide Most Significant Change stories as part of bi-annual project reporting.

Annex 1: Participant List

Name / Project / role / E-mail address
Boru Douthwaite / Facilitator /
Sanjini de Silva / Facilitator
Abdelbagi Ismail / PN 7 /
Thelma Paris / PN 7 /
Chu Thai Hoanh / PN 10 /
Nguyen Duy Can / PN 10 /
Aileen Lapitan / PN 11
Guanghui Xie / PN 16 /
Francis Bousquet / PN 25 /
Christophe Le Page / PN 25 /
Sawaeng Ruaysoongern / PN 28 /
Saiyon Sriyorach / PN 28
Francis Murray / PN 35 /
Masao Imamura / PN 50 /
Alyne Delaney / PN 52 /
Simon Cook / BFP /
Pamela George / CP Secretariat /
John Howell / CP M&E Consultant /
Kim Geheb / CPWFMekongBasin Coordinator /
Audrey Nepveu / IFAD /

1

[1] Boru Douthwaite, Senior Scientist, Rural Innovation Institute, CIAT, Cali, Colombia, e-mail:

[2] Sanjini de Silva, Head, Communication and Knowledge Sharing, IMWI, Sri Lanka, e-mail:

[B1]

[B2]

[B3]This vision PN10 has some grammar- typos going on- change for other? Or edit?

[B4]Typo?