REAP Presents: Training on Program Evaluation
Paul Glewwe (University of Minnesota)
August 22 & 23, 2009
Institute of Geographic Sciences and Natural Resources Research
No.11 Jia Datun Road, Chaoyang District, Beijing, China
Numerous programs are implemented by governments and non-governmental organizations (NGOs), which are intended to change individuals’ economic or social outcomes. Common examples of this include agricultural extension services, public health programs and education programs. An important (and admittedly difficult to answer) question is: How effective are these programs in changing economic or social outcomes? Comparing the relative effectiveness of different programs, as well as comparing these programs’ benefits to their costs is crucial for governments to understand.
Objectives
· To obtain a better understanding of how to objectively assess the impacts of programs through program design and data analysis
· To specifically discuss possible scenarios based on how intervention status is decided, and methods for analyzing data for each scenario
Agenda
These lectures present methods that can be used for different kinds of data, both randomized and non-randomized data. In the first lecture, an overview of the basic framework and methods for randomized experiments is presented.
While the simplest randomized experiments imply fairly simple econometric methods, things can quickly become more complicated. Lecture 2 provides detailed recommendations about how to design and then analyze data from randomized trials. Sample size, design and the power of experiments as well as practical design and implementation issues are covered.
Interventions are not randomly assigned in some studies, but instead conditional on some observed variables. Once certain variables are controlled for, there is no correlation between the treatment and the factors of interest. Within a subset (a certain value of the controlled variable), the treatment is randomly assigned. In Lecture 3, this scenario is further detailed, including discussion on regression methods, propensity scores, and estimating variances.
Another realistic scenario is when the treatment status still depends on the variables of interest, even after controlling for certain factors. In this circumstance, there are a number of econometric methods for evaluating programs that are often used, including: 1) Instrumental variable (IV) methods; 2) Regression discontinuity designs (RD); and 3) Difference in differences (DID) methods. Lecture 4 covers these, as well as two less commonly used methods, bounds analysis and sensitivity analysis.
Lecture 1: Basic Framework and Methods for Randomized Experiments
Lecture 2: Detailed Advice on How to Design, and Analyze Data from Randomized Experiments
Lecture 3: Estimating Program Impacts when Unconfounded Assignment Holds
Lecture 4: Estimating Program Impacts when Unconfounded Assignment Does NOT Hold