Useful Methodologies for Impact Evaluation of Aid for Trade
Abstract
According to the 2005Hong Kong Ministerial Declaration, Aid for Trade should aim to help developing countriesto build the supply-side capacity and trade-related infrastructure needed to expand their trade and to implement and benefit from the World Trade Organization (WTO) Agreements (WTO, 2006). Aid for Trade is expected to enhance growth prospects and through growth, increase income and reduce poverty (WTO, 2006). Stakeholders demand evidence of theeffectsof Aid for Trade and demand aid programs to becomemore results-oriented. Impact evaluation helps identify the effects of development aid, providing valuable insight and helping guide policiesby helping stakeholders understand what works and why.
Although there is this need to measureoutcomes and impacts when evaluating Aid for Trade, this has itschallenges (OECD, 2011). Attributing the effects to a program and calculating what might have occurred without the program is no easy feat. Evaluators are challenged to find comparison or control groups, take into account heterogeneous impact, establish attribution, and gather the necessary data. Useful methodologies can assist in dealing with these challenges when evaluating impact. Experts propose the use of different methodologies, each with its ownstrengths and weaknesses, for impact evaluation with no single agreed upon solution or guideline. Random Control Trials for example might produce more certainty in attribution but in many cases, is not practical or ethical because a control group which does not receive assistance must be selected and studied. Quasi-experimental methodologies are less likely to raise ethical concerns but rely upon the collection of baselinedata and risk internal validity, with only the ability to estimate causal relationships.
This research wasconducted in conjunction with the International Trade Centre (ITC), a joint agency of WTO and the United Nations. As an implementer of Aid for Trade, ITC’s mission is to “enable small business export success by connecting small and medium-sized enterprises to the global trading system”. This research will provide the ITC and its affiliates with lessons learned from existing impact evaluations and guide their policies and impact evaluation designs.
Aid for Trade covers a range of development activities including interalia policy support, access to finance, infrastructure development, and capacity building but this research has focused on those programs which are similar to most ITC programs. In order to offer guidance on useful methodologies for impact evaluations of Aid for Trade, the methods of six impact evaluation studiesof ITC-like programs, are critically assessed. Several interesting facts are discovered through this research. First, it confirms that very few impact evaluations of Aid for Trade have been conducted. Second, it shows that the diverse goals and context of the interventions call for the use of diverse and often a combination ofmethodologies. Third, it confirms data requirements for impact evaluation can create insurmountable challenges and limit the evaluation. Fourth, an evaluation which does not incorporate a methodology to rigorously establish attribution will result in a weak impact evaluation, which cannot be utilized for accountability or learning.
Choosing the appropriate methodology can make the difference between answering the question of the evaluation and not. Evaluation needs to solve the challenge of identifying comparison or control groups, handling heterogeneous impact, establishing attribution, and gathering the necessary datawithin a manageable budget. The research presented in this paper offers insight to help guide the choice of methodology for impact evaluation and ways to handle these challenges to increase our knowledge in this field and encourage further discussion, research, and improvements.
Table of Contents
Abbreviations and Definitions
1.Introduction
1.1Past Research on Impact Evaluations
1.2Research Questions and Approach
1.2.1Sub-questions
1.2.2Systematic Review Plan
1.3Relevance
1.4Delimitations
1.5Expected Challenges
2.Aid for Trade
2.1Aid for Trade Introduced
2.1.1What is Aid for Trade?
2.1.2Aid for Trade’s Emergence
2.1.3Rationale
2.1.4Aid for Trade Strategies
2.1.5Financing
2.2Stakeholders
2.2.1Donors
2.2.2Implementers
2.2.3Beneficiary
2.3The Objectives of Aid for Trade
2.3.1Expected Outcomes of Aid for Trade
2.3.2Objectives in Practice
2.3.3Aid for Trade in the Context of this Review
2.4In Summary
3.What Is Impact Evaluation of Aid for Trade?
3.1Evaluation of Development Aid
3.2Results Chain
3.3The OECD-DAC Evaluation Criteria
3.4Impact Evaluation Defined
3.5Why Perform Impact Evaluation?
3.6Why is Impact Evaluation of Aid for Trade so Limited?
3.7Is Impact Evaluation of Aid for Trade Difficult?
3.7.1Challenges with Attribution to Aid for Trade
3.7.2Constructing Control Groups
3.7.3Heterogeneous Impact
3.7.4Challenges Created by Changes over Time
3.7.5Availability of Data
3.7.6Additional Expense
3.8Guiding Principles for Impact Evaluations for Aid for Trade
3.8.1Validity
3.9What constitutes a ‘useful’ methodology?
3.10Wrapping Up
4.Impact Evaluation Methodologies
4.1Methodologies
4.2In Summary – The Methodologies
5.Research Method
5.1Search and Selection
5.1.1Unit of Analysis
5.1.2Methodology
5.1.3Search
5.1.4Selection Criteria
5.1.5Variables
5.1.6Challenges
5.2Limitations
5.3Determining Usefulness
6.Results Summarized
6.1The Selected Evaluations
6.1.1Evidence of Accountability and Learning
6.1.2Individual Program Evaluation vs. Aggregate-level Evaluation
6.1.3Timelines
6.1.4Quality of the Evaluations
7.Analysis of Results
7.1Addressing the Challenges with Impact Evaluation of Aid for Trade
7.1.1Attribution
7.1.2Heterogeneous Impact
7.1.3Formation of Control Comparison Groups
7.1.4Challenges with Timelines
7.1.5Collection of Data
7.1.6Cost
8.Conclusions
8.1How useful were the Methodologies for Impact Evaluation?
8.2The Importance of Objectives
8.3Recommendations for ITC
8.3.1Methodologies
8.3.2Design Considerations
8.3.3Attribution
8.4Future Research
8.5Final Words
Resources
Appendix A
Appendix B
Appendix C
List of Figures:
Figure 1- Illustration of Aid for Trade and its relation to national strategies
Figure 2- Aid for Trade by Region and Category (OECD and WTO Aid for Trade at a Glance Report, 2011)
Figure 3- Aid for Trade potential impacts
Figure 4 - PSDP Results chain Adapted from Kluve, Boldemann, and Weidnitzer, 2011: 30
Figure 5 - Results chain for results-based management (OECD, 2011: p. 80)
Figure 6 - Factors contributing to Women Empowerment
List of Tables:
Table 1- ITC Example Program Objectives
Table 2- ITC Activities in Aid for Trade
Table 3- Qualitative Methodologies
Table 4- Experimental Design Methodologies
Table 5- Methods for Creating Control or Comparison Groups
Table 6- Quasi-Experimental design methodologies
Table 7- Logical Counterfactual Methods
Table 8 - Exploratory Techniques
Table 9- Search Attempts - Summarized
Table 10- The Six Evaluations - An Overview
Table 11 - Evaluation focus – Program or Aggregate?
Table 12- Timeframes around the evaluation and interventions
Table 13- The Quality of the Evaluations
Abbreviations and Definitions
3ie International Initiative for Impact Evaluation
AEA American Evaluation Association
ALNAP The Active Learning Network for Accountability and Performance in Humanitarian Action
CEGA Center of Evaluation for Global Action
CIDA Canadian International Development Agency
DCED Donor Committee for Enterprise Development
DEReCOECD DAC Evaluation Resource Centre
DFID UK Department for International Development
DIME Development Impact Evaluation Initiative
EUEuropean Union
IAIA International Association for Impact Assessment
IDRC International Development Research Center
ILO International Labour Organization
IOCE International Organization for Cooperation in Evaluation
IPA Innovations for Poverty Action
IDB Inter-American Development Bank
ITC International Trade Centre
MDG Millennium Development Goals
NONIE Network of Networks for Impact Evaluation
ODA Official Development Assistance
ODI Overseas Development Institute
OECD The Organisation for Economic Co-operation and Development
SMEs Small and Medium Enterprises
UNCTADUnited Nations Conference on Trade and Development
UNEG United Nations Evaluation Group
UNIDO United Nations Industrial Development Organization
USAID U.S. Agency for International Development
WTO World Trade Organization
WFP World Food Programme
1.Introduction
Broadly speaking, Aid for Trade aims to help developing countries trade. Assistance is targeted to building supply-side capacity, trade-related infrastructure, policy and development (WTO, 2006). Official Development Assistance (ODA) flows in 2008 showed that 37 per cent of sector allocable ODA went to Aid for Trade programs and projects. Overall Aid for Trade commitments had increased to USD 39 billion in 2008 (OECD, 2011[2]: 22).[1] With this significant investment in Aid for Trade,and its recent official operationalization by the World Trade Organization (WTO) members,there is demand for evidence of impact. There is also a move toward making aid programs, especially those conducted by the United Nations organizations,more results-oriented. Evaluating impact helps in determining the effects of aid and provides valuable insight and helps to guide policies. Impact evaluation is undeniably an important field for researchers, development practitioners, and policymakers.
Literature suggests impact evaluations of Aid for Trade are limited and often face challenges in measuring outcomes and impacts (OECD, 2011) signifying there is still much to learn in this field. Attributing the effects to a program and calculating what might have occurred without the program is no easy feat. Evaluators are challenged to find comparison or control groups, take into account heterogeneous impact, establish attribution, and gather the necessary data. Useful methodologies can assist in dealing with these challenges when evaluating impact. Experts propose the use of different methodologies, each with its own strengths and weaknesses, for impact evaluation with no single agreed upon solution or guideline. Random Control Trials for example might produce more certainty in attribution but in many cases, is not practical or ethical because a control group which does not receive assistance must be selected and studied. Quasi-experimental methodologies are less likely to raise ethical concerns but rely upon the collection of baseline data and risk internal validity, with only the ability to estimate causal relationships.
This research will analyze impact evaluationsof Aid for Trade, critically assessing the methodologies used in the evaluation. The results of this research will provide guidance on useful methodologies for impact evaluations of Aid for Trade. This research is being conducted in conjunction with the International Trade Centre (ITC) in Geneva, Switzerland, a joint agency of WTO and the United Nations. ITC provides development solutions which enable small business export success in developing and transition-economy countries. High quality evaluations of the impact of ITC’s programs and services is essential to ensure they continuously improve, deliver value for money, and demonstrate impact. Increasing the impact of ITC’s technical assistance activities in developing countries and maximizing impact on enterprise competitiveness and export performance are priorities in ITC’s 2012-2015 Strategic Plan. ITC’s direction is also to incrementally demonstrate the socio-economic impact of its work.
Aid for Trade covers a range of development activities including inter alia policy support, access to finance, infrastructure development, and capacity building but this research has focused on those programs which are similar to most ITC programs. In order to offer guidance on useful methodologies for impact evaluations of Aid for Trade, the methods of six (6) impact evaluation studies of ITC-like programs, are critically assessed. Several interesting facts are discovered through this research. First, it confirms that very few impact evaluations of Aid for Trade have been conducted. Second, it shows that the diverse goals and context of the interventions call for the use of diverse and often a combination of methodologies. Third, it confirms data requirements for impact evaluation can create insurmountable challenges and limit the evaluation. Fourth, an evaluation which does not incorporate a methodology to rigorously establish attribution will result in a weak impact evaluation, which cannot be utilized for accountability or learning. This paper is divided into seven (7) chapters beginning with the introduction (Chapter 1) followed by an overview of Aid for Trade and ITC programs (Chapter 2), an introduction to impact evaluation (Chapter 3), and a discussion of popular methodologies (Chapter 4). Then the research methodology is presented (Chapter 5), the findings from the empirical review of the evaluations is offered (Chapter 6), followed by an analysis of the usefulness of the methodologies (Chapter 7) and conclusions (Chapter 8).
1.1Past Research on Impact Evaluations
Research on the methodologies of impact evaluations in humanitarian aid, microfinance and social programs is available. For example, David Hulme’s, Impact Assessment Methods For Microfinance reviews the impact assessment methodologies for microfinance (2000); the ALNAP’s,Improving humanitarian impact assessment: bridging theory and practice authored by Karen Proudlock and Ben Ramalingam examines four case studies of humanitarian impact evaluations in practice (2008);the Inter-American Development Bank’s,Do we Know What Works? authored by Cèsar Patricio Bouilon and Luis Tejerina provides results from a systematic review of 76studies of various social programs; and scholar (and economist) Howard White and others have published papers based on analyses of the methodologies used on individual aid impact evaluations (e.g. the 3ie Working Paper, Theory-Based Impact Evaluation: Principles and Practice (2009) and White, Sinha, and Flanagan, A review of the State of Impact Evaluation (2006)). In comparison, research on the methodologies for impact evaluations of Aid for Trade does not, to my knowledge, exist.
A 2011 OECD (Organisation for Economic Co-operation and Development) report from a meta-evaluation of 162 trade-related evaluations,[2]Strengthening Accountability in Aid for Trade, analyzing the outcome reporting of these evaluations (trade outcomes and impacts) but contributes very little new insight to our understanding of impact evaluation design. Although the title implies a focus on Aid for Trade, the majority of the evaluations included in the study were only partially focused on trade; just 42 evaluations were considered to have a more direct bearing on trade. Despite its attempt to claim otherwise, the report does not examine the methodologiesof impact evaluations, yet it makes conclusions about appropriate methodologies without backing this up with evidence. Leveraging a sample of this size could have offered the OECD the opportunity to learn something from existing methods but instead the report merely counts keyword use within the evaluation. Therefore, the value of this report is highly questionable and it certainly does not provide new information on methodologies of impact evaluation of Aid for Trade.
1.2Research Questions and Approach
Aid for Trade aims to help developing countries trade and is often targeted at building supply-side capacity, trade-related infrastructure, and trade development, policy and regulation. Some theory suggests that by supporting the expansion of trade, a resulting effect will be a reduction to poverty and a positive social impact and some Aid for Trade interventions even include these as objectives. And yet we cannot predict the impact of Aid for Trade since past research provides limited and conflicting information and because objectives, implementation, and external factors of each program are unique. Thus impact evaluations can be useful for examining the value of Aid for Trade interventions.
How is the impact of Aid for Trade evaluated? Impact evaluations can tell us which interventions work and don’t work as well as answering the questions about why they work. To obtain this information, various methodologies are currently used in the design of an impact evaluation. No single, accepted methodology is right for every situation and experts make various claims as to the best methodology for various conditions with little evidence to endorse their claims. This has led me to the following research question:
What methodologies are Useful for conducting impact Evaluations of Aid for Trade and under what conditions?
1.2.1Sub-questions
Several sub-questions help form chapters of the thesis and the basis for the research, contributing to the main research question. The sub-questions are:
1.What is Aid for Trade?
2.What is impact evaluation and what does it imply in the context of Aid for Trade?
3.When impact evaluations are designed, what methodologies can be used and when are they useful?
4.What methodologies are being used for impact evaluation of Aid for Trade?
5.What are the strengths and weaknesses of the methodologies in practice? How useful are these methodologies for each situation?
My approach to answering those sub-questions is outlined below. The sub-questions are in sequence and create discrete chapters of the thesis.
1. What is Aid for Trade?
This chapter (Chapter II) will tell the story of Aid for Trade, as the initiative triggered at the 2005 Hong Kong Ministerial Conference of the World Trade Organization (WTO). This includes a discussion of Aid for Trade’s emergenceand the rationale behind it including theories behind expected results. This chapter will include a discussion of the diverse objectives and designs of Aid for Trade and its many stakeholders. The aim of this chapter is to provide a clear definition of Aid for Trade, differentiating it from other forms of development assistance. Within this, I am most interested in identifying the qualities of trade assistance the ITC is delivering since this will guide my selection criteria for the empirical review.
Relevant discourse, interventions, evaluations, policies, strategies and theories of Aid for Trade will be analyzed. Sources will include the International Organizations involved with Aid for Trade initiatives (WTO, ITC, UNCTAD and the Aid for Trade Advisory Group) and various other institutions involved with Aid for Trade or similar initiatives such as recipient countries, donor aid associations, and foreign ministries.
2. What is impact evaluation and what does it imply in the context of Aid for Trade?
This chapter (Chapter III) will begin by defining ‘impact evaluation’ in the broader setting of evaluation of aid and discussing its rationale and principles. This will be followed by a review of the many possible objectives and the key elements of a high quality impact evaluation with a discussion of how this applies to the evaluation of Aid for Trade interventions. This chapter presents the challenges of evaluating Aid for Trade and helps define the elements which should be considered when choosing methodologies.