Improving UK NGOs’ effectiveness and transparency and shaping EU aid performance

Improving UK NGOs’ effectiveness and transparency and shaping EU aid performance
An evaluation of DfID’s 2011-13 grant to Bond
Ann Sanders
Jean Barclay
Janice Needham
September 2013

EXECUTIVE SUMMARY

Introduction

Bond is the UK membership body for non-governmental organisations (NGOs) working in international development. This is an evaluation to provide an assessment of the outputs, outcomes and impact of the work carried out under a programme of effectiveness, transparency and EU advocacy work undertaken by Bond. The programme, from 2011 to 2013, was mainly funded through an accountable grant for £543,520 from the Department for International Development (DfID).

The two-year programme of activities led by Bond, working with its members and peers, was intended to deliver the following outcome:

UK international NGOs are global leaders in assessing and transparently demonstrating their performance and contribution to poverty reduction, [and to] implement the International Aid Transparency Initiative (IATI) standard, and play a leadership role with European civil society in holding the EU to account against stated targets and commitments on the quantity and quality of its aid and on the outcomes of the EU Multiannual Financial Framework (MFF) negotiations.

·  45% of the funding was allocated to develop an Effectiveness Programme, comprising four main products: the Health Check, Impact Builder, Evidence Principles and Value for Money Paper.

·  21% of the funding was allocated to provide training, support and resources for NGOs to publish transparently the objectives and results of their work and in particular to publish to the IATI standard.

·  34% of the funding was allocated to support and represent UK NGOs in advocacy work on EU aid quantity and quality.

The objectives of this evaluation are to:

·  assess whether all grant activities have been completed as planned and have achieved the stated outputs and outcome and contributed towards the broader impact envisaged;

·  examine whether the grant represented good value for money and aligned with DfID’s objectives;

·  assess Bond’s role in delivering the programme, the added value it was able to bring, and to identify positive and negative aspects of its approach which could provide learning for the future.

Summary of key findings

Overall performance

·  The DfID grant to Bond has contributed significantly to UK international NGOs assessing and transparently demonstrating their performance and contribution to poverty reduction and implementing the IATI standard. In addition, Bond is highly respected within CONCORD and by other EU stakeholders and is recognised as having played a leadership role in holding the EU to account against stated targets and commitments on the quantity and quality of its aid and on the outcomes of the EU MFF negotiations.

·  Bond has enabled cross-sector collaboration and adds value through its convening power, its strategic reach and the access it provides to policy and practice. Bond is recognised as having led the way in developing new and effective approaches to effectiveness and transparency and has a strong reputation at a strategic level in the UK and EU.

·  Members have contributed time and resources to develop high quality products and support - including the first set of common indicators for the sector - that are relevant to a diverse range of organisations, although reach to smaller organisations may still need further consideration.

Programme objectives and outcomes

·  The programme objectives are still relevant and overall Bond has achieved the majority of the logframe outcome indicators and output targets. However, the considerable achievements from the grant are not adequately reflected in the narrow outcome measures identified in the logframe, which was developed at the outset of the programme. A closer focus on the theory of change in the structure of any future grant agreements may enable the development of more relevant indicators.

·  Most of the assumptions in the logframe are valid although there is insufficient evidence as yet to assess whether the assumption relating to the outcome in the logframe is valid, i.e. that UK NGOs’ willingness to use the common framework of indicators and data collection tools translates into a willingness also to report on performance publicly. Further research into barriers and support for NGOs in this area would be required to verify the assumptions and theory of change.

·  Any future programmes of this type will benefit from a greater focus on up-front planning on outcomes and developing appropriate indicators to enable more systematic and robust information gathering.

Effectiveness

·  Bond’s highly participative approach to engagement of NGOs has given credibility and a strong sense of collective ownership to the effectiveness programme, which is seen as innovative and ambitious. Feedback and evidence suggest effectiveness products are high quality, adaptable and making a difference to a range of types and sizes of organisation.

·  There is evidence that the products can bring efficiencies for both NGOs and funders.

·  As well as benefiting from the products themselves, NGOs have benefited from involvement in their development. Bond has created an enabling environment that brings organisations together to share information – including learning from failure – which has been highly valued. It is important that Bond seeks to develop further opportunities of this nature.

·  The logframe output targets for NGOs’ skills improvement and accessing support were achieved, but were not achieved for the implementation of the Health Check and Impact Builder, largely as a result of longer development timescales.

·  It would appear that the effectiveness products could be particularly beneficial to smaller and medium sized organisations, but that there may also be barriers to their ability to take full advantage of them due to capacity and resource constraints.

Transparency

·  Over 160 UK organisations have published to the IATI standard and the UK is seen as a leader in this area. Bond had provided 113 organisations with support at the end of June. NGOs benefiting are predominantly DfID grant holders.

·  NGOs are overwhelmingly positive about Bond’s transparency work and many have published more quickly and efficiently than they would have managed without Bond’s support. However, there is still work to be done to convince some NGOs of the benefits of publishing to IATI over and above the requirements of donor compliance, and of publishing data on performance and effectiveness.

Shaping EU aid performance

·  Bond is highly respected within CONCORD and by other EU stakeholders and is seen to have played a leadership role.

·  Outputs on aid quantity, quality and the Multiannual Financial Framework have been achieved and a range of stakeholders (UK NGOs, European NGO platforms and policy actors) have benefited in different ways.

Value for Money

·  The time and resources required to develop the effectiveness products has resulted in delay in their availability and as a result take up in some areas is lower than predicted. However, the contribution of Bond members to the process (and the time freely given) is an indication of the support and commitment from the sector and the value placed on Bond’s work.

·  There is evidence that the programme design was efficient, but delivery was adversely affected by staff turnover.

·  Bond used the grant equitably in relation to providing equality of access to resources.

Looking ahead

Areas for future development have been identified as:

·  a clearer articulation of the theory of change, with appropriate success indicators for all three strands of work supported by systematic information management and realistic delivery plans

·  supporting organisations to use the products and overcoming barriers to engagement, including reach to small organisations

·  testing and marketing of the products to maintain quality and ensure relevance and accessibility

·  continuing to provide an enabling environment for NGOs to share learning

·  supporting organisations to disclose data publicly

1.  INTRODUCTION

Aims and objectives

1.1.  Bond is the UK membership body for non-governmental organisations (NGOs) working in international development. This is an evaluation to provide an assessment of the outputs, outcomes and impact of the work carried out under a programme of effectiveness, transparency and EU advocacy work undertaken by Bond. The programme, from 2011 to 2013, was mainly funded by a grant from the Department for International Development (DfID).

1.2.  In 2011 Bond was awarded an accountable grant by DfID worth £543,520. The two year programme of activities led by Bond working with its members and peers was intended to deliver the following outcome:

UK international NGOs are global leaders in assessing and transparently demonstrating their performance and contribution to poverty reduction, [and to] implement the International Aid Transparency Initiative (IATI) standard, and play a leadership role with European civil society in holding the EU to account against stated targets and commitments on the quantity and quality of its aid and on the outcomes of the EU Multiannual Financial Framework (MFF) negotiations.

1.3.  The proportion of the total project budget for each strand of work is:

·  Effectiveness: 45%

·  Transparency: 21%

·  EU advocacy: 34%

1.4.  The objectives of this evaluation are to:

·  assess whether all grant activities have been completed as planned and have achieved the stated outputs and outcome and contributed towards the broader impact envisaged;

·  examine whether the grant represented good value for money and aligned with DfID’s objectives;

·  assess Bond’s role in delivering the programme, the added value it was able to bring, and to identify positive and negative aspects of its approach which could provide learning for the future.

1.5.  A follow-up exercise will be carried out after this evaluation to explore more fully:

·  features of the programme or the delivery process which could inform the second phase of the effectiveness and EU advocacy programme, funded by DfID for the period 2013 to 2016

·  issues arising from evaluating this programme of work which would be relevant in designing an evaluation of the second phase.

Methodology

1.6.  The methods used in the evaluation were:

·  A review of grant documentation to understand and attempt to articulate retrospectively the theories of change for each of the three elements of the grant – this is presented visually at Appendix A.

·  Development of key evaluation questions to provide a framework for semi-structured interviews and data analysis based on the theories of change for each part of the programme.

·  A review of the documentary evidence and data available from Bond reports and monitoring information (statistics quoted from Bond surveys are indicative as they are drawn from a relatively small sample size).

·  Semi-structured interviews with 34 stakeholders, including Bond member organisations, other INGOs engaged in the programme, DfID staff, EU stakeholders, European platforms and Bond staff. A list of interviewees is provided at Appendix B. 17 UK NGOs were interviewed including one Southern partner (Gender Links). Of these, 2 (12%) are categorised by Bond as small sized organisations, 8 (47%) as medium and 7 (41%) as large. This is a higher ratio of medium and large to smaller NGOs than that of Bond’s membership, which is largely a reflection of availability of personnel during the period of the evaluation (August and early September 2013).

1.7.  The grant objectives and targets agreed with DfID were structured around a logical framework (logframe), which in turn has shaped accountability requirements and information collection and thus the structure of this report. However, a significant limitation of the logframe is that several indicators do not fully reflect the main thrust of activities or outcomes. This evaluation therefore reports on the core elements of the logframe, but also aims to provide a wider picture of outcomes and learning over the grant period by structuring findings around the theory of change developed for the evaluation.

1.8.  Although funded from within the same grant, the work to support NGOs to perform effectively and transparently is distinct from the work shaping EU aid performance. The evaluation methodology has reflected this by developing separate theories of change and interview questions for each area of work.

1.9.  The effectiveness and transparency elements of the grant have supported the development of a range of tools, resources, activities and events. These are referred to generically throughout this report as “products”.

Context

1.10.  In its annual report to DfID in July 2012, Bond contextualised the issues it aimed to address with this funding. In summary these are:

·  UK NGOs struggle to provide rigorous and consistent evidence of the contribution they make to poverty reduction. This undermines performance-based decision-making, weakens learning and effectiveness, and ultimately feeds public scepticism towards international development. Bond and NIDOS research has consistently identified weaknesses in monitoring and evaluation as a barrier to improved effectiveness.

·  There is very little publicly available data or comparable information on the work of NGOs of the kind provided by publishing to the IATI standard – which countries they are working in, what the objectives and results of their programmes are, who is funding them and how much money they are spending. The lack of this data has two main negative consequences: it hampers NGOs’ own ability to be effective and deliver the best results for people living in poverty and a lack of open information prevents other stakeholders from understanding what NGOs do and being able to hold these organisations to account.

·  The EU plays a leading role in global development. Individual EU member states are also some of the world’s largest aid donors and some of the most influential global actors. The current difficult European economic context is limiting political space for ambition and progress on development. There is a risk that aid and development will fall off the political agenda in Europe. Bond members recognise that Europe, as the world’s largest aid donor, is an important political target for advocacy on development policy and practice. However at the same time, the majority of UK NGOs members do not have the internal capacity or resources to deliver either UK or Brussels-based EU work.

1.11.  The grant for the effectiveness work was awarded to Bond, with a small proportion of funds transferred to the Network of International Development Organisations in Scotland (NIDOS). The targets within the logframe for engagement and product use reflect members of Bond, NIDOS and the Coalition of Aid and Development Agencies within Northern Ireland (CADA). In addition to supporting the development of Bond’s products, NIDOS has developed its own Effectiveness Toolkit and some of the grant has been used to align the NIDOS and Bond products. Data in this evaluation has not been disaggregated by UK country network and the products reviewed are those developed by Bond.