Readiness Assessment

IMPACT MEASUREMENT

READINESS ASSESSMENT:

EAST AND CENTRAL

AFRICA REGION

Final report

Tom Barton

October 2010

1

Foreword

Colleagues,

In April 2008, CARE launched a timely initiative to move to a programme approach. At the heart of this ambitious strategy is the concept of “measuring our contribution to lasting social change”. In recognition that if human conditions are to improve in any sustainable way, working on MDG targets alone is insufficient. More fundamental issues that keep people in poverty, such as poor governance, discrimination, etc., must be addressed. We are faced with the challenge of measuring social change in a world that is increasingly critical of INGOs efficiency and accountability. Seeking a viable way forward, ECARMU decided to carry out an impact measurement readiness / capacity assessment will all of its COs. This report contains the results of a six months long research carried out in FY10 on our COs’ Impact Measurement capacities. Under the expert facilitation of CARE’s old friend and DM&E specialist Tom Barton, all COs’ DM&E, Knowledge Management and Programme Quality systems underwent a participatory assessment.

The results presented here are extremely rich and informative and, while pointing at existing good practice, also confirm important gaps between our current IM, DM&E, KM and PQ systems and practices from where we’d like to be. The task looks daunting but we can’t run away from it. The minimum we owe our impact groups and partners is to know when we are contributing to change and how. This report will enable us to develop a long term vision and a multi-year plan for a Impact Measurement competent DM&E system for our COs, where organizational learning and forward accountability are central. It will be a long term investment over several years and one that will require consistent leadership at all levels in an organizational environment characterized by competing priorities. We, the ECARMU Management Team, are committed to this process and will track progress on a regular basis.

At the same time that we will be working on regional and country specific issues arising from this study, we strongly urge that all CI members and other RMUs look at and reflect on these important findings. We have seen from Tom’s work extending this IMRA study in Malawi, Mali and Niger that very similar issues are also affecting countries in SARMU and WARMU. We have also been lobbied by representatives of CARE USA, CARE Norway, CARE Denmark and CARE UK to raise these issues with all CI members and seek ways to address the concerns as system-wide issues that require collective engagement and commitment. In that spirit, we will be sharing this report with many stakeholders beyond our corner of Africa.

We wish to thank Maliha Khan and Velina Petrova from CARE USA Program Impact unit for their important contribution to this process. Also, this work would not have been possible without CARE Norway’s financial support and feedback on the various products as they were emerging. We are also very grateful to COs for the time they invested in the assessment and resources they shared. Lastly, we wish to acknowledge the excellent work conducted by Tom Barton, considering the limited resources available for this task.

Onwards with impact measurement in ECARMU!

Regional Director or ECARMU Management Team

Impact Measurement Readiness Assessment: East and Central Africa Region October 2010 viii

Impact Measurement Readiness Assessment:

East and Central Africa Region

October 2010

Executive summary

Background and rationale

Recognising that addressing the underlying causes of poverty requires a long term commitment, CARE began in mid-2008 to shift from a project focus to a longer term programme approach. As a result of this change, CARE’s traditional ways of measuring results (by focussing on outputs) are no longer sufficient. CARE USA is now developing a new impact measurement strategy, intending to enable CARE and its partners to measure and track their contribution to creating sustainable change in the lives of specific impact populations of highly vulnerable people in poor communities.

Becoming competent in demonstrating impact is critical for all development stakeholders because of the increasing demands for accountability from donors, the public, partners, host governments and the communities being served. Doing so will, however, have major implications on the way CARE works and will require important organizational and personal change and commitment, including the allocation of sufficient resources. Seeing these challenges, the PQI and RMUs have advocated for a phased approach toward developing and implementing a new Impact Measurement strategy, i.e., working with a few “CO laboratories” to pilot-test elements prior to a global roll out. Meanwhile, the PQI and RMUs also recognise that there are many opportunities and good practices to build on from the region and elsewhere, and these need to be incorporated into the evolving IM strategy.

Objective of consultancy

The purpose of this consultancy was to carry out an impact measurement capacity/readiness assessment among ECA COs to inform the development (and eventual roll out) of the new CARE USA impact measurement strategy in the region in a way that will be meaningful, feasible and cost-effective to CARE USA but particularly to the ECA COs, their development partners, and their impact groups. More specifically, the impact measurement capacity assessment was designed to assess: 1) The ECA COs’ current/existing capacities to measure impact; and 2) What it will take to ensure COs’ readiness to effectively implement the new impact measurement strategy. This involved assessing the following in each CO: a) Existing M&E systems, processes and practices; b) Strengths and weaknesses of existing Knowledge Management (KM); c) Good practices; and d) Gaps to meet requirements of the new strategy.

Based on the above review, the study would formulate recommendations to the ECA COs, RMU and PQI about: 1) The feasibility and cost-effectiveness of the new CARE USA strategy and recommendations to increase both; 2) The types of systems and processes that need to be put in place at CO level in order to reconcile the different CARE International members’ needs and to ensure meaningful information is collected and analyzed at CO level against the set impact indicators; 3) The investment required to roll out the new impact measurement strategy in ECA, including the type of support and TA required from PQI and RMU; 4) Which COs are most advanced in their impact measurement efforts and could be used as laboratories for testing the new impact measurement system and as “resource centres” for others in the region.

Methodology

The IMRA was conducted through a combination of: a) self-assessment tools completed by the COs (using two tools – one a qualitative questionnaire on impact readiness and one a ranking tool on DM&E capacity); b) documents review of an extensive set of documents provided by each CO (related to IM, DM&E, p-shift, etc.), and site visits to all COs. The site visits included individual interviews and group meetings with key programme and programme support managers and staff. In most of the countries, it was possible to visit a field office for discussions at that level with project and programme teams, including some partners. The main contact persons in each CO were the ACD/P and the PQ/L (or senior DM&E) staff member. Limitations - there were delays in preparation for some countries, and a few offices had unique expectations of the exercise, e.g., being able to quickly solve problems related to DM&E or organisational structure. Although translation was needed (for French) in two countries, support was generally available to help with this.

The analysis was carried out with a qualitative content analysis approach, blending ideas from a guiding thematic framework included in the ToRs with emerging issues from the field. Preliminary analyses were done in each country and shared back with the CO teams in debriefing sessions within the CO visit, followed by brief summary reports based on the CO group work. The overall analysis was a rolling ground-based process, progressively enhanced over the duration of the study by the issues emerging from each of the countries as they were visited. A mid-term presentation of issues was done at the RPQG meeting approximately halfway through the set of CO visits. Attention has been paid to description of the situation, interpretation of driving factors where possible, and implications. A preliminary analysis of how to share and apply the results of this study was done with the DRD/PQ for ECARMU, and consideration of this issue is still on-going at the time of writing.

Summary of thematic issues and recommendations

Three major themes have emerged from the assessment of country offices readiness to step up the scale and quality of tracking impact as they make the shift to a programme approach. These themes have been sub-grouped to show issues, challenges, and resources/opportunities. The sub-themes are followed by recommendations, which came from the field and the consultant, and reflections on the implications of those recommendations.

The first of the major themes is DM&E capacity and competence, which is linked to the selection, collection and analysis of data to produce information. The self-assessments of the COs tended to rate their performance in this area as ‘moderate’, at least for output information, but there were many gaps in performance around tracking and analysing outcome/effects and impact level data/information. A particularly important concern is the tendency to mystify impact, regarding it as a numerical product, and only able to be demonstrated by highly technical baselines and endlines.

The second major theme is KM capacity and competence, which is linked to the production, capture and use of knowledge within the COs and their programmes, and thus, within the organisation and its partners. Although there was some consideration of this issue in the original ToRs for the IMRA exercise, so many issues and challenges arose that it can be regarded as an emergent area of strong concern. It is also a system area that is under-developed in CARE, compared to many other systems. The most critical concern in this area is the complete absence of any KM strategies or policies in all COs (and at other levels – RMU and CUSA).

The third major theme is PQ/L strategies and tools, which is linked to the production and application of credible evidence. Ideally, this evidence would be built on good/best practices, quality data and information, and relevant documentary and experiential knowledge. There are, however, significant gaps in quality assurance and reflection measures affecting DM&E and KM, and therefore, the quality of evidence about impacts of CARE’s work. Of particular concern is the need for holistic, systems-oriented capacity building support to programme quality, including DM&E, KM, and IM.

Emerging recommendations and implications

The recommendations arising from this IMRA have been articulated to address the IM challenges of the ECARMU COs, plus the RMU and CUSA as it relates to them. Meanwhile, there are several strands of evidence indicating that the same or similar issues pervade IM competency for all of CARE’s COs, including IMRA exercises conducted during 2010 in Malawi (SARMU) plus Mali and Niger (WARMU), as well as earlier work like the DM&E situation analysis in 2007 for the pan-Africa DM&E seminar series, and the global PQ meeting in 2008 (Istanbul) to launch the programme approach. The universality of these IM issues should compel global attention by all RMUs and all CI members within CARE International if CARE is going to move ahead effectively to track its impact.

The following core recommendations are overarching; they span all of the analysis of strengths, challenges and recommendations:

·  CARE International, including all CI members, RMUs and COs to reflect on the issues and recommendations of this analysis followed by strategic prioritisation and resourcing of systemic capacity building toward IM competence as an organisation

·  CO PQ/L teams, with ECARMU & programme partners to develop coherent and systemic CO capacity building plans for enhancing IM (and its related system components of M&E, KM and PQ/L) that will include holistic attention to building on strengths, addressing gaps and making improvements in tools, skills/knowledge, staff/infrastructure, and organisational structures/systems/roles.

The key recommendations for Theme 1, DM&E capacity and competence, are to:

·  further operationalise an international menu of outcome and impact indicators;

·  develop a template for programme (& CO portfolio) DM&E plans;

·  develop a menu of impact assessment methods (quantitative and qualitative) with guidance on use;

·  develop operational guidelines for on-going situation analysis and testing / reviews of programme ToCs;

·  enhance CO understanding on how to measure analyse contribution (vs attribution); and

·  improve basic DM&E skills, including urgent attention to building capacity in some COs and strengthening in others for reliable participant & output level tracking and reporting system.

The principal implementers of these recommendations should be CUSA & ECARMU, but working with COs. The implications of these recommendations include: a) provision of IM related TA, including capacity building for supervisors and managers; b) modification of JDs; and c) promoting more CO dialogue with donors, governments and other boundary partners.

The key recommendations for Theme 2, KM capacity and competence, are to:

·  develop functional draft CO KM strategies through participatory work in each CO;

·  promote a regional KM COP;

·  review all CO & CARE policies that affect or compromise good KM;

·  ensure adequate infrastructure for long term storage; and

·  generate an annual report from each CO that reflects on emerging evidence about impact.

The logical implementers of these recommendations should be CUSA, ECARMU and COs (especially a PQ-L/KM team). The implications are likely to include: a) provision of TA to facilitate system thinking and quality for developing KM; b) some budgeting, e.g., for hardware or internet connectivity; and c) ensuring careful operationalisation and good documentation in managing data and information for consistency and usability.

The key recommendations for Theme 3, PQ/L strategies and tools, are to:

·  examine review processes at all levels and develop appropriate guidelines to promote impact-level reflections in those events;

·  review and upgrade new staff orientation to address IM, DM&E, KM, and PQ/L;