WORKSHOP REPORT (Final)
Regional Workshop on Results Oriented M&E
Main Issues and Conclusions
Cape Town, 28-29 November 2004
The Evaluation Office organized a Regional Workshop on Results Oriented Monitoring & Evaluation for countries in the African region in Cape Town from 28 to 29 November 2004. This regional workshop forms an important part of the assessment of M&E practices and the build up towards a corporate policy on evaluation. Participants in the workshop included Deputy and Assistant Resident Representatives, Programme Advisers and M&E Focal Points from UNDP Country Offices as well as representatives from the Governments of Nigeria and Zimbabwe. The workshop had the following main objectives:
- To mainstream results oriented M&E and to reinforce the links between M&E and development results.
- To promote a dialogue on the evaluation function in UNDP and the UN system and to further refine evaluation methodologies.
- To identify demands, practices and lessons learned on M&E for informing the formulation of corporate country policy on evaluation as well as a strategy on capacity building.
- To interchange experience and practices on monitoring and evaluation among country offices, government and other stakeholders.
In preparation for the formulation of a corporate country policy on evaluation, an assessment of M&E practices is in process, to ensure that the future policy is grounded in genuine practices and needs on the ground in country offices. Consultations are also being conducted at different levels. Some actions so far have included:
- Outcome evaluation review November 2003 – March 2004
- Sarajevo and Kuala Lumpur workshops February 2004
- EvalNet survey July/August 2004, other EvalNet discussions
- Country evaluation workshop in New York November 2004
- Internal assessment of country M&E practices in selected countries ongoing since September 2004
Some preliminary findings from this assessment were discussed at the Cape Town workshop.
The attached concept note and agenda outlines the scope and background for the workshop in more detail. The following summary highlights key discussions, issues and conclusions from the proceedings of the workshop, while detailed presentations made and papers presented by participants are attached.
1. Opening Session
Following opening and welcoming statements by Mr. Thulani Mabaso, on behalf of Mr. Isaac Chivore, OIC of UNDP South Africa, and Ms. Saraswathi Menon, EO Director, three rich presentations discussed the broader and increasingly complex backdrop and context for development at the national, regional and global level, and UNDP’s role in this picture. They further posed some challenges for the monitoring and evaluation function. Some of the key points are summarized below:
The Deputy Chief of UNDP’s RegionalCenter in Johannesburg, Mr. Joseph Mugore’s presentation highlighted the following points with respect to Africa’s development context:
High growth rates are observed but at the same time poverty is spreading and deepening.
A number of externally defined national economic management schemes have been introduced over the years but national ownership has been questionable. To address this, there is, among other things, need for a strong state.
The links between democracy, freedom and development need to be further explored and seen in relation to aid dependency.
Involvement of civil society is important but NGOs are not always representative bodies. Genuine participation and ownership of the broader civil society and private sector would eliminate problems with capacity and absorption constraints. Consequently, with appropriate policies, delivery and implementation issues are less important.
The MDGs are disputed goals, some say they would not be achievable in Africa, but in reality they would be achievable with an appropriate paradigm shift. Both resource gaps and capacity gaps exist but could be addressed with a paradigm shift.
The President of IDEAS, Mr. Sulley Gariba’s keynote address raised the following points:
UNDP has been key to introducing a people-centered approach to development.
The MDGs have created an opportunity for a compact between developed and developing countries, and for an adequate distribution of roles.
On monitoring and evaluation, there is a strong need to further involve citizens, civil society, legislative bodies and other that so far have not been sufficiently involved. There should be genuine interaction among various groups of stakeholders, ensuring that public policies and private sector relate to each other. A key question to ensure substantive accountability is who commissions evaluations.
With changing participation in M&E, tools and capacity building approaches need to change as well.
The scope of M&E needs to go beyond aid, and UNDP has already taken initiatives in this direction. Imbalances between national institutions and traditional, external development institutions need to be leveled out.
The Country Director for UNDP Sierra Leone, Ms. Nancy Asanga made the following comments as a discussant:
Important changes are under way in Africa, including a higher level of demand for ownership.
Participation and capacity development of local partners are important, for development processes as well as for M&E specifically.
With Sierra Lone as an example, the integration of M&E into the inter-agency Consultative Group process has proven successful.
The EO Director, Ms. Saraswathi Menon, outlined the perspectives from the Evaluation Office as follows:
At present, there is no UN system wide function for evaluation. The emerging UNDAF evaluation mechanisms will be an important step in this direction. The United Nations Evaluation Group (UNEG) is also working on common norms and standards for the future.
Evaluation in UNDP has achieved institutional ownership, but a requisite independence of the function has been retained at the substantive level. There has been an important modernization of evaluations, going to a more upstream, results based model, but there is still some way to go in terms of establishing a culture of assessment and fully engaging senior management, in order to ensure that M&E is utilized to its full potential in strategic management and decision making. One of the key issues in this regard will be to ensure that adequate capacity is in place at all levels.
In moving towards an evaluation policy, three key questions need to be asked, for which input from the COs is welcomed: Further defining the purpose and role of evaluations, achieving balance between accountability, learning and management needs; establishing the right balance between compliance and optional approaches; and setting the appropriate objectives for evaluations, i.e. whether we are evaluating for processes or results.
Key issues raised during the plenary session included:
Broad overview of shifts in development and evaluation paradigms- from concern with inputs and activities to focus on results and development effectiveness
Sea change in the South- greater demand for accountability and results by both government and civil society. Key challenges for M&E is how to promote and support development effectiveness, ownership and capacity development and meet the need for “professionalization” of the field and greater and more effective coordination and harmonization
Tensions between aid effectiveness and development effectiveness
High growth rates in Africa over the last few decades but this has not translated into higher human development indices. In some cases paradox of high growth rates and fiscal discipline alongside deepening poverty.
Value of national planning –allows setting agendas in programme countries but SAPs and PRSPs have eroded and replaced this function and in most cases weakened already weak states in setting the national development agenda
Policy choices rather than growth or democracy on their own determine and/or influence poverty reduction.
Emerging global and national consensus on what constitutes development- human development, human security and lately MDGs. UNDP key contributor in shaping fundamental shifts in development thinking to human centered development and a more multidimensional notion of what constitutes development (human poverty , human security etc)
Compact between developed and developing countries on the need to forge partnerships, policy coherence and promote country ownership.
Fundamental questions for M&E are: What to evaluate, who commissions evaluations, who evaluates and how to evaluate ( methods and approaches)
Bridge asymmetrical relationships and broaden constituencies beyond Ministries of Finance e.g. new independent public evaluation institute.
Emerging trends point to the need for evaluations to target and mobilize citizen participation and parliaments which offer potential for strengthening accountability.
Outcome based evaluations entail need for partnerships
2. Approaches to M&E at Country Level
Two strong presentations of actual case studies were made by the Uganda and Tanzania country offices as follows (copies of presentations are attached):
(i)Ms. Rose Ssabatindra, M&E Focal Point, UNDP Uganda;“Using M&E to assess progress towards the MDGs and PRS(P).” This presentation introduced the PEAP, Uganda’s version of the PRSP, and discussed the use of RBM approaches and strategic M&E in the public sector reform programme. UNDP plays an important role, among other things in leading one of the MDG working groups.
(ii)Mr. Amon Manayema, Leader, Poverty Unit, UNDP Tanzania;“Monitoring and Evaluation of Poverty Reduction Strategies in Tanzania.” This presentation discussed the comprehensive system of working groups and surveys around Tanzania’s PRSP, and how this has strengthened the collaboration between Government and donors on strategic issues.
Both presentations pointed to the opportunities and potential for greater national ownership , coordination and harmonization , reduced transaction costs as well as the development of mutually agreed ( governments, sub national entities , citizen groups, donors ) comprehensive national M&E frameworks and tracking systems with clear and specific target setting, bench marking and focus on development results. Both also pointed to the complexities and process heavy systems now in place to monitor PRSPs/PEAPs, the parallel reporting systems and noted how budget/resource needs , ( a short term horizon ) often detracts from the focus on development results and national ownership.
The discussion following the presentations addressed, inter alia, the potential for UNDP to build national capacities of government entities and citizen groups to build credible M&E systems that track higher end development results ( e g. MDGs ) ( In the absence of such an approach there may be risk of marginalization and/or lack of alignment with national M&E frameworks of government and donor partners.)
3. Group-work: Assessment of Current M&E Practices at the Country Offices
Participants formed three working groups and each was asked to address one of three pre-designed questions to assess the current M&E practices in the country offices and their linkages with those of other partners, including national governments and institutions.
The following is a summary of key points presented by the groups to the plenary:
Key Issues – Group 1
The group discussed the following topic:
How do M&E instruments link up with overall government strategies?
The topic was discussed at three levels and in three categories - the M&E instruments, problem/challenges and solutions that are summarized below.
Some observations under the instrument category were:
Some countries have an audit system at national and local levels
Household Surveys and National Census exist
Monitoring Institutions exist
Regular Reviews and Reporting are undertaken
UNDAF and MDGs are good instruments
PRSPs / National Development Plans/ MTEF are available for M&E
NDRs and MDG Reports are a source of evaluative evidence
Public Hearings by Parliament and Parliamentary Committees could contribute to M&E
Problems / Challenges:
Different UNDP & Government Review and Reporting Cycles
Different Government and donor agency mechanisms (who is driving ?)
Poor capacity
Poor harmonization within the UN system
Not all national plans have and M&E component
UNDP can only realistically and strategically link with certain government systems
COs do not “reach out” sufficiently
Who owns the process? Are we concertedly building ownership?
Authenticity of baseline data? – gap between the development status quo and the desired outcome
Poor “inclusion” i.e. development effectiveness (poor ownership resulting in UNDP building M&E systems for itself).
Solutions:
Joint/Consensual UNDP/Government decision-making meetings ensuring government is driving the process
Joint assessments at programme design stage and joint evaluations during and after implementation
Use of ROAR as an instrument for programme management between UNDP and Government – increased collaboration through pro-active engagement based on empowering government leading to better harmonization
Build credible self-assessment systems through harmonizing national and local indicators with MDGs
Setting mutual agreed and realistic targets using proxy indicators aligned to MDGs
Investing in capacity building to address specific M&E problems by agreeing on common problems, instruments and systems
Outcome instruments to be brought “in line” with existing PRSP and budget support indicators
Bring “visibility” of the M&E function closer to the Country Level.
The discussion also focused on clarification of how the ROAR could be used as an instrument for harmonization. The group members explained that the ROAR is more of an instrument for management actions, not an M& E instrument per se.
Key Issues – Group 2
The group discussed the following topic:
In the broader context of the national development situation (e.g. MDGs, Governance Reform, HIV/AIDS, PRS(P), Direct Budget Support by bilateral and multilateral donors), how does UNDP’s result-based M&E practices reconcile the tension between institutional compliance and assessing development results?
Key issues presented by the group were:
RBM tools need to introduce accountability and oversight.
The ATLAS Project, Country Programme outcomes, SRF, MYFF and Core Results should also introduce accountability oversight.
Country Office Monitoring and Evaluation Team to be established under the leadership of a member of a senior management for Learning/Information sharing and enhancement of knowledge. The arrangement to result in effectively undertaking reporting, accountability, and oversight functions.
Country Office monitoring and evaluation processes and systems should link with government/NGOs/CBOs and other partners.
The M&E regime does not dovetail well into the new corporate tools and processes being introduced. There is need for the Bureau and EO to work on this.
There is need for the M&E function to be supported by the audit function.
The presentation provided a base for intensive discussions and views on the structure of M&E in the office. Various models were presented including DRRs' functions having M&E office functions, Senior National Officer heading M&E functions in the office, integrating functions in staff job descriptions. The meeting left the arrangements to the individual country offices.
Key Issues – Group 3
The group discussed the following topic:
What are the most significant challenges for CO to mainstream results-oriented M&E?
The following were identified as major challenges for country offices to mainstream results-oriented M&E:
How to harmonize different systems and tools of various partners including the national governments
Shift of monitoring and evaluation paradigm resulting in complexity of M&E concepts as it relates to looking at results and the need for a common understanding
Internal and external capacity building
Keeping all stakeholders/partners at the same level of understanding.
Internal and external capacity building.
Developing evaluation culture in CO.
Time required for results to be evident.
Developing M&E capacities of staff (day to day monitoring) M&E model – dedicated M&E person.
Make sure M&E is demand – driven making budgeting for M&E mandatory.
Resource mobilization for ECD.
Elaborate mechanism for M&E follow-up.
Ownership of the M&E process.
Soft intervention M&E methodologies/indicators.
Linking country offices based results into global M&E framework.
4. Wrap-Up of Day 1 Proceedings: Main Issues and Conclusions
A presentation was made by Ms. Fadzai Gwaradzimba, Senior Evaluation Advisor, EO, summarizing the main issues and conclusions discussed during Day 1. The text of the presentation is attached. Some of the main points highlighted were:
- The presenters formulated key challenges for M&E and development work globally, regionally as well as at the country level.
- There is a “sea change” taking place in the South, in terms of ownership and policy development, and UNDP is challenged to keep up with this change.
- The weakening of the state is a key issue to be addressed. Important policy choices lie ahead, among other related to the relationship between democracy and development.
- Some gaps do exist in the current M&E tools and systems. Some key questions include: What capacity exists within our current network? How does UNDP support national processes and systems? There is a need to close the gaps between national and international agendas and capacities.
5. Networking for capacity building in M&E: The case of the Niger Network
Presenter: Jean-Charles Rouge, M&E Specialist, UNDP Niger
The session focused on how ReNSE has been contributing to sustainable evaluation development capacity (ECD) in Niger and the extent to which UNDP’s involvement in ReNSE activities has increased its comparative advantage in the field of national ECD and M&E of PRSPs.
Key issues:
Regional Context:
- Development evaluation practices in Africa changed during the 1990’s
- Lack of development effectiveness despite large amounts of capital inflows in LDCs
- Inadequate “parachute” approach to evaluate development interventions: need to adopt an “ownership” approach, focusing more on knowledge transfers to national counterparts and national capacity building
- Increased need to have national M&E expertise to fully participate in evaluation missions
- Increased need to organize existing national expertise in order to strengthen its capacities
- Constitution of national M&E communities in Africa
- UNICEF led the way in helping the creation of several national & regional M&E networks/associations: Kenya (1997), Rwanda (1998), AfrEA (1998), Niger (1999).
- Today, more than 20 networks are active at national level and registered as AfrEA members.
- Using Internet and ICT to foster evaluation capacities: With little funding available, ICTs were the primary tools used to build these networks. A critical mass of people having access to Internet allowed them to start concrete ECD-oriented activities for their communities.
Niger