Learning Network on Capacity Development
Discussion paper 2013/02
Measuring Results
for Effective Institutions
Cristien Temmink
15 February 2013
Draft for discussion
What do we know?
As capacity development becomes mainstreamed in international development assistance programmes, demand for the evaluation of capacity development initiatives is growing. (Horton 2011:5) However, despiteall efforts and investments made and priority given to capacity development, assessing progress and achievements in capacity development remains a challenge faced by all players involved, be it civil society organisations (CSOs), official donors or state actors. (Watson 2010; Simister and Smith 2010)
Horton (2011: 5) notes that ‘while evaluations are now routine, they seldom satisfy either donors or programme managers.’He identifies four principal challenges facing the monitoring and evaluation (M&E) of capacity development:
- Capacity development is a complex change process
- Poorly designed capacity development interventions
- Weak design of capacity development evaluations
- Limited professional knowledge and expertise
Linked to thesechallenges, some key debatesregarding the measurement of capacity development (CD)are around:
- The purpose of measurement: accountability versus learning? (why measure?)
- The level of measurement(what, where and when to measure?)
- Approaches, tools and methods:results-based versus systems thinking (how to measure?)
The way in which capacity and capacity development are perceived (as a simple-linear-deliverable or a complex-unpredictable-emergent-endogenous process) has implications for how to go about its measurement. Often perceptions are not made explicit, leading to interventions that lack clarity and complicating the design of evaluations. Most CD is complex, often with unclear and uncertain links between external interventions and the ultimate outcomes. This makes the debate of what constitutes ‘results’ at what levels and moments in time,and how to measurethem, even more difficult. Questions regarding the perception of capacity, the purpose of CD, the purpose of measurement, and the level of measurement (what, where, when and with whom?) should inform the approachused.
Although there seems to be agreement that CD is a complex change process, M&E systems often focus on capturing relatively easytomeasureresults for accountability purposes. This could be explained by the fact that funders typically want clear-cut results, and by a desire to keep M&E simple and not too time-consuming. A rigid focus on results often hinders and obstructs the CD process and is counterproductive in terms of development effectiveness, but ‘mixed approaches’ to M&E may offer the possibility of measuring both quantitative and qualitative results while also supporting learning.
Challenges
Capacity developmentis a complex change process
Capacity development processes are inherently complex and their results unpredictable. (Horton 2011) Capacity tends to emerge over time, affected by many [internal and external] factors (Watson 2010), and is an endogenous process, often with uncertainty about the exact influence and effects of external efforts. (Horton 2011) As capacity development is associated with multiple causes and multiple actors influencing the process at different levels and at different moment in time, it is a non-linear process with no direct link between cause and effect. The extent to which capacity development efforts evolve into better performance in terms of improved development results is often uncertain, poorly understood, and unlikely to happen within the time frame of a project cycle. The time span between capacity development interventions and desired end results can be very long. (Simister and Smith, 2010)
As a consequence, capacity development processes cannot be neatly planned and implemented with predefined and predictable results. Tracking change is also difficultas developing capacity involves circles of learning through trail and error and applying lessons learned in the next cycle of activities. This complexity of capacity development processes and the emergence of unexpected results due to numerous unpredictable influences pose significant challenges for evaluators. (Horton 2011)
Poorly designed capacity development interventions
Complex change processes such as CD ‘cannot be neatly planned and implementedwith predictable results’ (Horton 2011);attempts to do so often lead to poorly designed interventions. Two main weaknesses in planning can be identified. First, a lack of clarity about what is meant by ‘capacity’ and ‘capacity development’ for a given context and set of stakeholders. This includes questions as to the purpose of CD, types of capacities to be developed, and who the providers and recipients are. (Simister and Smith, 2010) Watson (2010) observes that behind any discussion of ‘M&E of capacity’ [and capacity development] lies the challenge that ‘capacity’ is not yet a well defined area of practice.
Second, many programmes lack a credible programme theory or theory of change which makes explicit how the programme is expected to work, how it proposes to bring about its results, and what levels and whose perspectives we are interested in. (Simister and Smith, 2010; Horton, 2011) ‘Many organisations consider [capacity development] a fundamental part of what they do, yet very few understand what it is in a strategic and operational manner. They sense intuitively what it is. They know they do [capacity development] and why it is important (and spend large sums of money on doing so) yet they rarely conceive of it, operationalise it, or measure it in a way that helps them learn and improve their approach.’ (Ortiz and Taylor 2008: 24 in Simister and Smith, 2010)
In particular in complex change processes, ‘having such a theory is essential for programme operators (and evaluators) to learn from experience, by comparing expectations with actual results and reflecting on the differences between them.’ (Horton 2011: 6) From an M&E point of view, theories of change should not be very complex, but they should at least include a clear vision of what an intervention is trying to achieve, why, how, and what the eventual results might be. (Simister and Smith 2010)
Weak design of capacity development evaluations
Lack of clarity regarding the purpose of an evaluation (why evaluate?) combined with an absent or implicit theory of change (what to evaluate?) may result in weak design of capacity development evaluations. A particular challenge is that evaluations attemptto addressmultiple questions and purposeswithin a single evaluation carried out over a short period of time and with limited resources. (Simister and Smith, 2010) However, different types of evaluations are needed to answer different evaluation questions: learning- and improvement-oriented evaluations are designed and conducted differently from accountability-oriented ones. Learning-oriented evaluations have to engage programme managers, staff and beneficiaries in participatory exercises and often the evaluation process is more important than producing a report. Accountability-oriented evaluations, which seek to assure external stakeholders that resources have been well used and that the programme has generated significant results, are best carried out by external evaluators (often measurement specialists) who operate at arm’s length from programme personnel and the intended beneficiaries. Horton (2011)
Trying to combine too many questions and different purposes often leads to confusion and unsatisfactory evaluations, because when the foundation of an evaluation is feeble (and questions such as programme theory, purpose of the evaluation, and focus are unclear), it will be almost impossible to define which M&E approach to use, with its corresponding methods and tools, or to determine the time and resources needed.
Limited professional knowledge and expertise
Limited knowledge sharing and professional development and learning opportunities regarding M&E of CD is another challenge. There are some guidelines for evaluating CD (e.g. World Bank, UNDP), but there is little information on experiences with their actual use and results obtained. Not many evaluation reports are made publicly available, and there are no text books and few training programmes that include modules specifically on evaluation of CD. (Horton 2011) As a result, organisations often lack staff with expertise in M&E of CD, and get stuck in the challenges dscribed earlier. Watson (2006) also notes that development banks and official donoragencies face obstacles to improving their capacity development capacities due to lack of clarity about what measurement is for, lack of capacity, and inadequate practical guidance for staff.
Traps and debates
The purpose of measurement: accountability or learning?
A central debatearound M&E of CD (and of development interventions in general) is related to the purpose of measurement. Different actors have different reasons for measurement. On one hand, donors and development agencies strive to demonstrate the effectiveness of their funding – they need to show ‘results’ for accountability purposes.On the other hand, NGOs and development practitioners are interested in learning from experience as a means to self-improvement. (Watson 2010)In particular, when accepting that capacity development is a complex change process, learning becomes essential for dealing with its unpredictable nature and the emergence of results.
These different purposes for measurement lead to different information needs. Combining accountability and learning is therefore not a simple task, due to the different types of information required and the different methods and processes to collect it. Also, behind the different information needs, there are likely to be competing demands and interests that are difficult to reconcile and which often lead to real tensions between different stakeholders. (Simister and Smith 2010) Moreover, M&E for learning can be hindered when it is linked to accountability for funding decisions. ‘If people feel funding or their jobs are threatened they will be less likely to provide honest and open opinions about capacity and any changes resulting from specific interventions.’ (Simister and Smith, 2010: 7) A risk of accountability-oriented evaluations is that they tend to show only positive results, thus becoming artificial and meaningless, or even untrue.
As different stakeholder groups expect different things from evaluations, a trap may be aiming for ‘multi-purpose evaluations’ that try to please everybody, but in the end fail to satisfy anybody. Therefore it is important to recognize the need to conduct different types of evaluation for various types of user groups and purposes. (Horton 2011) There is an inherent trade-off with the two distinct objectives of accountability and learning. Power dynamics exist and should not be ignored, making negotiation and openness between stakeholders necessary at the outset. (James 2009)
There are also discussions about what constitute results. The European Centre for Development Policy Management (ECDPM)study on ‘endogenous’ capacity development (see ECDPM 2008), advocates for a balanced approach to M&E of CD that combines accountability and learning by acknowledging the ‘soft’ sides of capacity and performance as legitimate and essential development results. (Engel et al, 2007)
Although studies emphasise the importance of M&E for learning to improve, accountability to donors appears to be dominating evaluation, which ‘has not become mainstreamed as a learning tool or a management practice aimed at improving programmes.’ (Horton 2011: 6) Taking different (political) interests, perspectives, demands, as well as power dynamics and funding sources into account, it remains a question remains whether M&E (of CD) really can serve two masters at once. (Simister and Smith, 2010)
The level of measurement
Another question in the M&E of CD interventionsis the change that different stakeholders are looking for and where this change is supposed to happen: at the level of the recipient organisation/ institution, their clients/beneficiaries or even the wider community?
Debates around this often stem from the fact that stakeholders ‘hold differing views on capacity [and capacity development] and as a result act with different goals mind.’(Engel et al., 2007)While some notions of capacity merely imply performance improvement in the sense of being better able to deliver predefined outputs, ECDPM identified other important features of capacity, summarised in five core capabilities. In addition to ability to produce development results, they also include the abilities to create operating spaces and sound relationships; to self-organise and act; to create coherence and direction; and to learn and adapt to changing circumstances over time.
A basic distinction in this discussion is whether capacity development is a means to an end (strengthening organisations or systems to perform or implement defined activities) or an end in itself (strengthening organisations or systems to fulfil their missions or rolesas defined by themselves). (James 2009; Simister and Smith 2010) In the first case, results are likely to be sought at the level of the organisation/institution and within the time span of the CD intervention or shortly after. In the second case, results are sought in de wider system (final beneficiaries, community or society) and would take more time to ‘ripple out’. The ECDPM definition sees capacity as both a means to improve performance and as an end in itself: ‘capacity is that emergent combination of attributes, capabilities and relationships that enables a system to exist, adapt and perform’. (Watson 2006)
Donors often insist that it is not enough to measure only the direct impact of the CD intervention, but that the contribution itmakes to wider development goals such as poverty alleviation should also be measured. CD providers on the other hand, argue that all they can realistically assess is change at the level of their client organisations. Measuring how these changes ripple out is highly problematic in terms of the extra investments needed, issues of access to beneficiaries and the fact that changes in the ultimate beneficiaries are subject to many more contextual influences than simply the capacity development intervention. Trying to attribute change at wider levels is problematic, tenuous and costly. (James 2009)
Is it enough to assess changes (results) within the specific project (often at the organisational or institution level) to show improved capacities, or should wider results spreading out in time and space be ‘measured’? (Simister and Smith 2010) Another question to address is what is technically possible and sound in terms of ‘measurement’ at different levels. Simister and Smith (2010) argue that evaluations should differentiate between changes that can be measured and others that can only be illustrated. Plausible links between measurable changes and wider goals may enable M&E to be more realistic and less burdensome in terms of time and resources. Moreover, they warn to be cautious not to promise results that cannot be delivered, thus creating unrealistic expectations in proposals.
Approaches, tools & methods:results-based versus system thinking
Discussions aboutthe purpose of measurement (in particular the drive for accountability or learning) are pushing M&E of CD in two different directions: one reflects a traditional results-based, logframe approachto intentional change, while the other relies on an open systems way of thinking, each with its related M&E methods. (Watson 2010)
The logical framework approach is based on alogic of linear ‘cause and effect’ relationship between input, outputs, performance and development goals and is often used to focus on delivery of pre-defined outcomes. This is also the basis of a Result-Based Management (RBM) approach. There has been a growing international call for results-based management, whereby development actors are asked to be accountable for and demonstrate achievement of “measurable” results (Paris Declaration, 2005, Accra Agenda for Action, 2008 and High-Level Forum on Aid Effectiveness in Busan, 2011, Open Forum for CSO Development Effectiveness in Istanbul, 2011). Many organisations try to strengthen their PME systems in response to this call. (Ongevalle et al., 2012: 3)Also development banks and donors tend to use the project (or logical) framework as their design tool and for monitoring progress and evaluating effectiveness. The main reason they appear to favour these approaches is that they provide the basis for meeting accountability needs towards policy makers, politicians and taxpayers. Watson (2006)
Critics observe that after more than two decades of implementing a results agenda, success stories remain limited. Development actors continue to face problems in implementing results-based management approaches in a way that contributes to improved analysis, planning and decision-making. Instead, they are often mechanically used for accountability and control purposes. (Vähämäki et al, 2011 in Van Ongevalle 2012)Also, the need to show results can lead to risk-averse behaviour and focus on results that are more tangible and easy to measure. Consequently, organisations that work towards less tangible change, such as capacity development, find themselves struggling to measure results using established monitoring and evaluation tools (Stern et al. 2012 in: Van Ongevalle 2012).
In the context of capacity development, the debate is about whether results-based approaches are helpful in capturing these complex change processes or not. Opponents argue that results-based approaches are technocratic, reductionist, control-oriented, linear, and unable to capture complexity. Also, since they monitor progress in relation to pre-determined indicators, theyattract attention away from ‘less tangible and more relational/attitudinal dimensions of capacity and from broader learning from experience. In many cases unanticipated results or insights may prove more important to development effectiveness than what was planned.’ Watson (2010: 241)Others note that using results-based models can undermine CD as it forces programme managers to deliver on pre-established results regardless of changing contexts and circumstances. (Horton 2011) Another problem is that formalised results-oriented systems may require so much effort that they divert resources from an organization’s primary mission. (Watson 2006)
Systems thinking and related approaches are believed to have much to offer to the design, management and evaluation of CD interventions.When CD is considered multi-dimensional in nature (as in the ECDPM framework), efforts to enhance organizations’ capacities are not linear, but interact with living systems. CD is associated with multiple causes, solutions and effects (some unintended), and interaction between stakeholders is important but is often not controllable and potentially unpredictable. A consequence is that detailed performance (or capacity) improvement plans are not easy to make as capacity tends to ‘emerge’ over time affected by many factors. (Watson 2006)Methods such as outcome mapping are able to capture this complexity as they offer a non-linear, vision driven approach with a focus on changes in behaviours, relationships, and actions of the people and organisations involved. Other techniques and methods used are Most Significant Changestories, Appreciative Inquiry, Client Satisfaction Instruments and Horizontal Evaluations. (Simister and Smith 2010; Horton 2011)These methods are participatory, qualitative and learning- and improvement-oriented.