Spanish Impact Evaluation Trust Fund
2008 Annual Report
TABLE OF CONTENTS
Executive Summary and Recommendations 1
I. SIEF Evaluation Funding 2
II. SIEF Capacity Building 6
III. SIEF Knowledge Dissemination 7
IV. SIEF Management and Staffing 8
V. SIEF Financial Update 8
Annex 1: Quick Wins Window (US$ 1.5 million) 10
Annex 2: Cluster Fund Window (US$ 6 million) 29
Annex 3: Innovation Fund (US$ 2.6 million) 45
Annex 4: Building Capacity 47
Annex 5: Sharing Results and Knowledge 50
Annex 6: TORs for SIEF Positions 53
Annex 7: SIEF Financials 65
Spanish Impact Evaluation Trust Fund Annual Report 2008
Executive Summary and Recommendations
This report reviews SIEF’s first 15 months of operation, highlighting the substantial role the initiative has already played in shifting the paradigm inside and outside of the World Bank in the quest for aid effectiveness, and drawing lessons from the initial year of experience that can guide SIEF’s future implementation. The report includes an executive summary followed by annexes highlighting the three funding windows, capacity building, knowledge management, and financial reporting.
The Spanish Impact Evaluation Fund (SIEF) which became effective in September 2007, is the largest trust fund focused on impact evaluation ever established in the World Bank.[1] A €10.4 million program funded by Spain, SIEF’s goal is to capitalize on the World Bank’s global engagement in social sector investments to build an evidence base on what works to improve human development (HD) outcomes through rigorous, scientifically valid, impact evaluation. SIEF pursues this goal through three lines of activity:
i) Financing impact evaluations – SIEF provides grant financing to prospective impact evaluations in eligible developing countries;
ii) Training - SIEF builds technical capacity for evaluation through regional workshops in the developing world; and
iii) Knowledge dissemination – SIEF supports publication and web-based dissemination to promote the use of evaluation results to inform policy.
SIEF supports research in 11 thematic areas carried out in countries in every region of the developing world, supports regional capacity building for conducting applied impact evaluations and supports the creation of a global knowledge base on development effectiveness.
In the space of 15 months since the creation of SIEF, the program has begun to deliver on its core objectives and is now at a stage where it is ready to shift from design and funding allocation to implementation and knowledge generation.
To date, SIEF has achieved the following outputs:
· Financing of 50 high quality, prospective impact evaluations, representing approximately one-third of all impact evaluations supported by the World Bank in the field of human development.
· Training over 600 developing country officials, World Bank staff and bilateral and multilateral development agency staff in impact evaluation methods.
· Supporting the publication of Conditional Cash Transfers: Reducing Present and Future Poverty (Fiszbein and Schady 2009), a book synthesizing the global evidence on conditional cash transfer programs, based principally on the meta-evaluation of a series of rigorous impact evaluations.
· Supporting the publication of several research articles, World Bank staff working papers, and a rich set of web-based materials.
Looking forward, additional outputs and some final results can be expected as the impact evaluations directly supported by SIEF move into implementation. For 2009, the program is on track to produce 20 baseline data sets and accompanying analyses, eight papers, five regional training workshops and numerous dissemination activities. It is projected that by the program’s forecasted closing in December 2010, SIEF will have contributed to 25 policy papers, 15 regional impact evaluation training courses, and numerous dissemination activities.
The final result of these efforts will be threefold: (i) building an empirically valid evidence base to inform development policy, particularly in the six strategic areas supported by the SIEF clusters; (ii) building the capacity of a new generation of practitioners and policymakers in developing countries who will be both contributors to and users of impact evaluations to inform policy decisions; and (iii) contributing to knowledge generation and sharing on development effectiveness, notably among communities of practice in the area of human development.
SIEF has claimed attention in the global media, with articles in both The Economist and the Financial Times citing the innovative character of impact evaluations it is supporting. Finally, SIEF is credited within the World Bank for institutionalizing a powerful new model to support evidence-based policy, with small, expert teams managing “clusters” of evaluations in key policy areas aimed at generating operationally relevant evidence on how programs work under different country conditions.
I. SIEF Evaluation Funding
SIEF’s premier objective is to promote development results and aid effectiveness by advancing global knowledge of “what works” to promote HD outcomes through rigorous impact evaluation. It has funded evaluations through three distinct windows: i) Quick Wins ii) the Cluster Fund and iii) the Innovation Fund. While their objectives are somewhat different, all three funding windows require a rigorous technical review – which is a hallmark of the SIEF program -- based on four major criteria:
- Strategic importance for policy.
- Technical rigor and robustness of evaluation design.
- Capacity of evaluation team, including track record on quality products and timely delivery.
- Alignment of the proposed evaluation with World Bank and Spanish donor priorities.
The initial SIEF experience has also attracted US$ 1.5 million in cofinancing from the UK Department for International Development (DfID), which was favorably impressed with the program’s high bar for technically rigorous evaluations and for the quality of its regional training programs, in which DfID staff consistently seek admission.
Quick Wins Window (US$ 1.5 million). This window was designed to circumvent the long lead time required to launch and complete rigorous impact evaluations by channeling funding as early as possible to already designed or ongoing, high quality evaluations which had a funding gap. This enabled SIEF to get a head start on demonstrating the kind of products that results-focused donors quite justifiably demand before continuing or expanding support for new programs. During the first SIEF Steering Committee meeting in July 2007 (before the SIEF Trust Fund became effective), it was agreed to grant US$ 1.5 million in funding to 13 high quality evaluations prioritized by the SIEF Technical Committee.
As of December 2008, 60 percent of the total grant had been committed or disbursed, the majority of this concentrated in 10 of the 13 evaluations which had committed or disbursed the majority of their allocated funds (please see the annex on financial implementation for details).
By end-2008 the Quick Wins program had helped generate evaluation evidence that:
· the “conditions” in conditional cash transfer programs don’t always matter – in Ecuador, families randomly selected to receive the transfer without any conditions that they send their children to school still did so at higher rates than families not receiving transfers.[2]
· a low cost program to hire community teachers in rural Kenya in order to lower class size boosted learning achievement among first and second graders by 12 percent.[3]
· community sanitation drives in rural India to promote the use of latrines were successful in increasing their use (from less than 5 percent to over 30 percent of villagers, on average) but had disappointingly little impact on children’s health (almost no reduction in diarrhea)[4];
· the results of a social protection program in Chile targeted to the extreme poor initially confirm sustained gains in intermediate indicators health and education; and
· a new youth employment program in Honduras required significant modification if it was to reach its target audience of low-income teenagers with low formal education, evidence the government has embraced and is using to redesign the program.[5]
Quick-wins funding also supported several ongoing evaluations expected to be completed in 2009, including a highly anticipated study of the long term effects of early childhood nutrition interventions in Jamaica. The funding allowed the research team to track beneficiaries who are now aged 22 - the longest-term follow up of an early childhood development (ECD) program ever undertaken in the developing world. This SIEF-supported research is expected to contribute to the highly important but scarce literature on the long-term impacts of pre-school education that is at present concentrated in the US and Europe.
The Quick Wins window also allowed five new evaluation proposals that had not found funding to move forward. While these are still at least a year away from completion, these evaluations will generate important results before the end of 2010. Finally, Quick Wins funding permitted timely collection of baseline survey data during 2008 in Uganda Youth Employment, Tanzania HIV/AIDS, and Morocco conditional cash transfers (CCTs). Two others cases, Jamaica Early Childhood Development and Lesotho HIV/AIDS, have initiated work that is expected to be complete by the end of 2009.
However, not all of the Quick Wins evaluations proceed smoothly. Three of the 13 evaluations (Mexico early childhood development, Ecuador conditional cash transfer and Brazil early childhood development) suffered significant delays and one (Benin school based management) secured alternative sources of financing. Therefore, under the terms of the Quick Wins window, these monies have been reclaimed and US$ 439,000 is currently available for redistribution as will be discussed in the upcoming SIEF Steering Committee Meeting scheduled for March 3, 2009.
The degree of attrition encountered in the Quick Wins round is instructive, as these evaluations were mostly in an advanced state of development at the time of funding (July 2007), and the raison d’être for their inclusion as “quick wins” was their promise of early results. These lessons should be kept in mind in setting expectations for evaluations financed in the Innovation and Cluster Fund windows.
Additional information on the Quick Wins window is available in Annex 1.
Cluster Fund Window (US$ 6 million). The majority of SIEF evaluation funding - US$ 6 million of the US$ 9 million total – is allocated to impact evaluations within thematic “clusters” in key areas of human development (HD) policy. The cluster model takes advantage of the World Bank’s position as the principal global development institution to support the generation of an evidence base on development effectiveness generated from rigorous impact evaluations. The Cluster Fund evaluations create an opportunity for researchers working in a particular policy area to collaborate and share information on methodology, instruments, research design, and results over the long-term. These studies create a platform upon which cross-country analysis can be conducted to build the base for meta-analyses.
Cluster Fund leaders play a key role within the thematic cluster area by facilitating information sharing, building the global evidence base, and supporting researchers and policymakers within thematic communities of practice.
Through an extensive, bottom-up process managed by SIEF in 2008 during which 160 evaluation proposals were submitted, six priority clusters were identified in January 2008:
· Health Contracting/Pay for Performance.
· Conditional Cash Transfers/Second Generation Issues.
· Malaria Control.
· Active Labor Markets/ Youth Employment Programs.
· Basic Education Accountability.
· HIV/AIDS Prevention.
Following the selection of the clusters in January 2008, teams leading selected impact evaluations within the clusters were invited to submit full concept notes which are subject to a rigorous peer review process.
As of December 31, 2008, less than one year after cluster funds were identified, of a total of 26 cluster fund evaluations, 18 have submitted concept notes, 15 have been approved for funding and two are pending revision. Eight evaluations have begun to disburse and commit funds for a total of US$ 676,000.[6]
Although most Cluster Fund evaluations are just now getting underway, each of the eight evaluations underway has collected baseline data and generated baseline data reports. The majority of the remaining 26 evaluations supported by the clusters are on schedule to generate baseline data and produce reports on initial results by the end of 2010.
Additional information on the Cluster Fund evaluations is available in Annex 2.
Innovation Fund Window (US$ 2.6 million). To complement the SIEF support to clusters of evaluations, the US$ 2.6 million Innovation Fund window was established on the premise that it is also important to generate evaluation evidence on promising, new innovative approaches with the potential of scaling up. Wholly new approaches by definition cannot yet be the subject of parallel evaluations in different country contexts. The Innovation Fund window thus provides support to individual impact evaluations of programs with high demonstration potential. In every case, the research must focus on interventions, programs and approaches that affect Human Development (HD) outcomes, even if the intervention or program is not in a HD sector. As one of the objectives of the SIEF is to understand the scope for non-HD interventions and programs to improve HD outcomes, the Innovation Fund made awards through separate HD (US$ 1 million) and Sustainable Development (SD) (US$ 1.6 million) windows. Four HD and eight SD impact evaluations were funded for a total of US$ 2.69 million.
In the six months since the Innovation Fund awards were announced, six evaluation teams have submitted concept notes, one of which has completed the peer review process and is accessing its SIEF funding. The remaining six are currently finalizing concept notes.
Additional information on the Innovation Fund is available in Annex 3.
II. SIEF Capacity Building
SIEF’s priority agenda includes conducting up to five regional workshops per year to provide customized training in impact evaluation fundamentals. The training is targeted to joint teams of government officials, local researchers and World Bank task teams who are working together on the design of new projects. The goal is to equip these teams with the knowledge and technical skills needed to build high quality impact evaluations into the design of these new projects, making it possible to collect baseline data and establish a valid comparison group from the outset of the program and to generate just-in-time, operationally relevant advice to support program implementation.
The workshops are divided into two tracks – one geared toward individuals with a strong background in the methodology, which is called the technical track, and the other policymaker track, geared toward those participants who may be new to impact evaluation. With SIEF funding, the HD Chief Economist’s unit has been able to refine and scale-up the impact evaluation training model it began developing in 2005.