Evaluations of NGO International Development and Humanitarian Work

Evaluations of NGO International Development and Humanitarian Work

Christina Laybourn for Bond

This briefing paper offers a concise overview of current practices and debates in the field of NGO evaluations, particularly of development and humanitarian work. Whilst it is by no means comprehensive, it aims to present pertinent issues and stimulate discussion. The first section provides an overview of evaluation mechanisms, methodologies, and quality initiatives. The second section summarises some of the most common challenges experienced by NGOs conducting evaluations, drawing on comments from NGO professionals, websites and research articles. Finally, there is a short bibliography of existing toolkits and standards, and relevant reading, where the issues can be explored in more detail.

Evaluation Practice

Evaluation Mechanisms:

As monitoring and evaluation has become increasingly used by NGOs around the world, many mechanisms have been developed to facilitate the process. Some of the most common approaches include:

• Performance indicators
• The logical framework approach
• Theory-based evaluation
• Formal surveys
• Rapid appraisal methods
• Participatory methods
• Public expenditure tracking surveys
• Cost-benefit and cost-effectiveness analysis
• Impact evaluation[1]

There are now a wide-range of training workshops and toolkits to help NGOs implement these M&E strategies, however, evidence suggests that implementing an effective M&E system takes considerable time and support[2]. As will be discussed later, there are still considerable challenges to NGO evaluations, however some NGOs have developed internal evaluation standards to ensure that their assessments are of a high quality. For example, Oxfam GB has developed Programme Evaluation Standards which outline Oxfam’s policy on how to conduct programme evaluations, and how to respond to and share evaluation findings[3].

Evaluation Methodologies

Self-assessment

As confidence and training in M&E methods grows, NGOs are increasingly choosing to conduct evaluations of their work themselves[4]: asking staff to assess programmes or projects, and submit reports to senior management. Self-assessment has the considerable advantage of being cheaper to implement than hiring an independent consultant. It also can encourage learning, and a sense of ownership of the evaluation process amongst staff. Agency staff may also be better placed to understand the complexities of a project or programme than an external evaluator.

However, NGOs can find that self-assessments can be difficult to conduct effectively, and internal evaluators may miss some things that an outside observer would pick up on. They also lack a guarantee of impartiality. For these reasons self-assessments can have less credibility.

Peer-assessment (also know as joint evaluation)

Peer-assessment involves staff from one or more peer organisation conducting the evaluation. This means that, again, costs are kept relatively low, however evaluations by people external to the organisation can be more insightful, and more credible. Because the assessors come from similar organisations, they will be well informed about the challenges of the sector.

The principle difficulty with peer assessment is that it requires a high level of trust between NGOs, so that the assessed organisation can be open and honest during the evaluation. Where there is strong competition for funds, for example, NGOs may be unwilling to let a potential rival organisation scrutinise their operations, in case the information is later used against them.

Large NGOs sometimes opt for an approach halfway between the two above: sending staff from one country office to evaluate the work of another office. This combines both the benefits of familiarity with the organisations systems, and the insight of an outside perspective.

Peer-review

A slightly different approach is where an NGO submits their self-assessment to a peer organisation for review, or where a peer conducts an assessment of an NGO’s evaluation system. As will be discussed later, this is an effective way of maintaining evaluation standards, but does not actually involve the peer assessing the NGO’s work.

External-assessment

An NGO may commission an independent consultant to conduct an assessment of some or all of its work. Although external-assessment is increasingly giving way to self-assessment, it is still often required by donors when large amounts of funding have been awarded. External-assessments have more credibility, and, as with peer assessments, an outside perspective can offer some useful insights and suggestions, that the organisation’s staff may not have thought of.

However, external assessments can be very costly: where the overall programme budget is small to begin with, hiring an independent consultant may simply not be feasible.

Quality Assurance

As will be discussed later, one of the primary difficulties cited with NGO evaluations is their variable quality. Where evaluations lack reliable detail or analysis, their findings will not be credible or useful. Several quality assurance initiatives have been developed in an attempt to address this, although they are primarily intended for donor bodies, rather than NGOs themselves.

The OECD Development Assistance Committee (DAC) has produced a set of Principles for evaluating development assistance, intended for donor governments and multi-lateral organisations. First published in 1991, the Principles include impartiality, independence, credibility and usefulness. These Principles are now widely adopted by development agencies in DAC member countries, and by several multilateral/international financial institutions, such as UNDP, the World Bank, and the IMF. Members of the DAC conduct joint evaluations of peer members against the Principles. DAC has also recently published a set of Quality Standards for Development Evaluation, which are intended to provide a guide to good evaluation practice. The Standards follow the structure of a typical evaluation process: defining purpose, planning, implementing, reporting, and learning from evaluation results.

Although the DAC Principles and Standards are intended for DAC members, they can be adopted by other development partners. For example, the Humanitarian network ALNAP has adapted the DAC Principles for conducting evaluations in complex emergencies, into a guide for Humanitarian agencies. DFID’s guidance notes for NGO evaluations also draw on the DAC Standards.

The UN Evaluation Group has also developed extensive guidance for conducting evaluations, including Standards, a Code of Conduct, Norms and Ethical Guidelines. All UN departments now conduct evaluations according to these standards, meaning that evaluations from different departments can be collated and presented collectively to donors.

Several donor agencies have systems in place to ensure the quality of their own evaluations, or those of their funded NGOs. For example, the Swedish development agency, Sida, commissioned an independent assessment of Sida evaluations in 2008, which assessed their coverage, credibility and usefulness of the results information. The report found several areas of weakness, and recommended improvements to Sida’s evaluation processes[5].

The Dutch Ministry of Foreign Affairs’ Evaluation Department (IOB) similarly conducted an assessment of the Ministry’s new Tailor-made Monitoring system for NGOs, to assess its benefits and disadvantages[6]. The Dutch Ministry also asks NGOs in receipt of the largest grants to submit their self-evaluations to the IOB so that the quality of the evaluations can be assessed.

ALNAP has made notable efforts to monitor and improve the quality of evaluations by Humanitarian NGOs, beyond adapting the DAC Principles, as mentioned above. Between 2001 and 2006 ALNAP conducted yearly “meta-evaluations” of Evaluations of Humanitarian Aid (EHA), using a Quality Pro Forma that they developed according to accepted evaluation good practice. Details of the “meta-evaluations” were published in ALNAP’s yearly Review of Humanitarian Action. This autumn, ALNAP has begun a process of action research focusing on strengthening the evaluation capacities of humanitarian organisations, and has produced a draft paper and framework of how to improve the effectiveness and use of humanitarian evaluations[7].

Common Challenges with Evaluations

It has widely been recognised in the literature and by NGO professionals, that there are considerable difficulties with conducting evaluations. Some of the most common challenges are discussed below.

Poor Quality of Evaluations

A principle concern cited by donors and researchers, is that evaluations can differ markedly in their quality[8]. Poor quality evaluations fail to provide the reader with any true or reliable insight into the achievements, or challenges, of a programme, and are essentially useless. The most common reasons for a poor evaluation include: difficulties gathering data, lack of critical analysis, lack of contextualisation, insufficient time allowed, and inexperienced evaluators[9]. This can be due to the complexities of the programme context: data gathering can be difficult and unreliable in remote or unstable situations. Insufficient resources, as discussed below, can mean less time for training. DFID’s Johnathan Patrick also highlights that a lack of NGO specific standards or “meta-evaluations” is a principle failing in ensuring the quality of evaluations[10].

Lack of Resources for Evaluation

Responding to concerns over the quality of evaluations, NGO professionals counter that they lack the resources to conduct or commission sufficiently in-depth evaluations[11]. The pressure on NGOs to keep operating costs down means that M&E budgets are often sqeezed. Short term funding cycles are also cited as counterproductive to long-tern reflective evaluation of programme impacts. However, arguably, both funders and NGOs may be guilty of underestimating how much time and money is required to conduct a satisfactory evaluation.

NGO professionals also argue that the monitoring and evaluation requirements of many donors are inappropriately burdensome, and are often not proportional to the size of NGOs or the size of the grant awarded[12]. Furthermore, the workload for NGOs is compounded by donors asking for different information in their evaluations[13]. It has been suggested that it would be more efficient for donors to agree on a universal set of requirements for all evaluations by NGOs.

Accountability and Learning Paradox

Another of the common problems cited, is the sometimes contradictory purpose of conducting an evaluation. Evaluations are intended to highlight both the successes and challenges experienced during a programme of work. However, evaluations are also often commissioned at the request of donors. NGO professionals see this as creating dual pull between learning from mistakes, and accounting to your donor[14]. If future funding is in any way influenced by the NGO’s previous achievements, there are clear incentives to emphasise successes, and minimise failures. However, by not reporting or examining any challenges, valuable learning opportunities are missed, both by the subject NGO, and by others who read the evaluation.

Use of Evaluations

The ALNAP report on the utilisation of evaluations, notes concern in the sector that evaluation results will not be used[15]. Jonthanan Patrick, Evaluation Adviser at DFID, recognises this concern amongst NGOs that evaluations will “fall into a black hole” once they are submitted to a donor agency[16]. The sense that submitting evaluations to donors is merely a “check box” exercise, and that they are not properly read, may contribute to the varying standards by which evaluations are conducted. Donor agencies such as DFID and the Dutch Ministry of Foreign Affairs admit that they cannot yet adequately process and collate all the findings from evaluations that are submitted to them[17]. However, there are some efforts to collate findings from evaluations: the Norwegian development agency NORAD has commissioned synthesis reports of evaluations of Humanitarian assistance, and Environmental Development assistance[18].

Learning from Evaluations

One of the most concerning issues around evaluations of NGO work, is that the findings are not shared, and recommendations are not implemented. Although evaluations are widely perceived to function as a tool for learning, an ALNAP report found very little evidence that evaluations are used by NGOs to bring about changes or improvements in operations[19]. An evaluation of the quality of DFID’s evaluation reports and assurance systems similarly noted a problem with top management having an unduly defensive attitude to recommendations in evaluations, and suggests that there is a need to move towards a work culture which acknowledges past weaknesses and welcomes lesson learning.[20]

Efforts are also being made in the NGO community to improve learning from evaluations between NGOs: ALNAP recently published a draft framework to improve utilisation of humanitarian evaluations[21], and the South Asian umbrella group Community of Evaluators, is currently holding a conference in New Delhi, focusing on how to learn from evaluations.

Evaluation Bibliography

Tools for Evaluation

Monitoring and Evaluation Resource Guide, CES

Evaluating Humanitarian Action Using the OECD/DAC Criteria: an ALNAP guide for humanitarian agencies, ALNAP

Evaluating Development Co-operation: Summary of Key Norms and Standards, OECD DAC

Programme Evaluation Policy, Oxfam GB

Monitoring and Evaluation: Some tools, methods and approaches, The World Bank

American Evaluation Association Guiding Principles for Evaluation, AEA (2004)

Standards for Evaluation of the UN System UNEG (2005)

Program Evaluation Standards: Raising the Accountability Bar, InterAction (2009)

References and useful reading

Jean Ellis, Accountability and Learning: developing monitoring and evaluation in the third sector, CES (2008)

Marta Foresti with Christina Archer, Tammie O’Neil and Richard Longhurst A Comparative Study of Evaluation Policies and Practices in Development Agencies Agence Française de Développement/ ODI (2007)

Alistair Hallam Strengthening organisational capacities for evaluation of humanitarian action (Draft) ALNAP (2010)

Hilary Nalven ‘How Do We Measure What Really Counts?’ InterAction’sMonday Developments (September 2009) pp38-9

Brian Pratt, RethinkingMonitoring and Evaluationin ONTRAC 37 pp1-2

Peta Sandison, ‘The Utilisation of Evaluations’ in ALNAP Review of Humanitarian Action 2005, ALNAP (2005) pp89-144

Rachna Sundararajan ‘Making a Diifference: confidence and uncertainty in demonstrating impact’ InterAction’sMonday Developments (June 2008) p24

Kate Wright What Role for Evaluation in the Context of Performance-Based Management? INTRAC (2008)

1

[1] More details, and the advantages and disadvantages of each approach can be found here: Monitoring and Evaluation: Some tools, methods and approaches, The World Bank

[2] Jean Ellis, Accountability and Learning: developing monitoring and evaluation in the third sector, CES (2008)

[3] Oxfam GB Programme Evaluation Policy

[4]Monitoring and Evaluation in the Third Sector: Survey Findings Charities Evaluation Services (2007) p2

[5] Kim Forss, Evert Vedung, Stein-Erik Kruse, Agnes Mwaiselage, Anna Nilsdotter Are Sida Evaluations Good Enough? An Assessment of 34 Evaluation Reports Sida (2008) DESC&departmentid=298&topheight=55&headerheight=23&fotheight=0&leftframewidth=300&width=820&stylesheet=sida.css&frameout=0&language=14&login=True&username=sih

[6] ‘Maatgesneden monitoring, beperkte beleidsdoorlichting MFS’ (A Limited Policy review of MFS Tailor-made Monitoring) (2009)

[7] Alistair Hallam Strengthening organisational capacities for evaluation of humanitarian action (Draft) ALNAP (2010)

[8] Jonathan Patrick, private conversation 26/10/10; Ellis (2008) p9

[9] Jean Ellis Monitoring and evaluation in the third sector: meeting accountability and learning needs Paper presented at the 15th NCVO/VSSN Researching the Voluntary Sector Conference 2009

[10] Jonathan Patrick, private conversation 26/10/10

[11]Rachna Sundararajan ‘Making a Diifference: confidence and uncertainty in demonstrating impact’ InterAction’sMonday Developments (June 2008) p24

[12] Blomeyer and Sanz, Survey of ECDG funded NGOs (2010); Ellis (2008) p6

[13] Peter Konijn, Cordaid, private conversation 14/08/2010

[14] Summary Report: European M&E Workshop INTRAC 2005; p 8; Harry Derksen, ICCO, private conversation 12/08/2010

[15] Peta Sandison, ‘The Utilisation of Evaluations’ in ALNAP Review of Humanitarian Action 2005, ALNAP (2005) p90

[16] Jonathan Patrick, private conversation 26/10/10

[17] Jonathan Patrick, private conversation 26/10/10; Phil Compernolle, Evaluator, Policy and Operations Evaluation Department, Dutch Ministry of Foreign Affairs, private conversation 03/09/10

[18]Humanitarian Response to Natural Disasters: A Synthesis of Evaluation Findings, NORAD (2009); A Synthesis of Evaluations of Environmental Development Assistance by Multilateral Organisations, NORAD (2009).

[19] ALNAP (2005)

[20]Rodger C. Riddell The Quality of DFIDs Evaluation Reports and Assurance Systems IACDI (2009)

[21]Hallam (2010)