STRATEGIC IMPACT EVALUATION FUND (SIEF)
IMPACT EVALUATION CLUSTER NOTE:
BASIC EDUCATION SERVICE DELIVERY
I. Background.
School systems in most developing countries largely fail to deliver quality primary education to poor children. Although considerable gains have been made in bringing children into school, there are still regions in the world where primary enrollments and retention remain low; currently an estimated 60 million primary school-age children are out of school. But the bigger problem may be that even when children are in school, they are learning very little. The results of international achievement tests consistently show that even children who do enroll attain such poor instruction that they are effectively functionally illiterate.
Until recently, efforts to improve education typically focused on providing inputs to schools—increasing spending along existing allocation patterns. But, substantial research now demonstrates that increased funding is not sufficient for improved learning outcomes. First, incremental funds may be allocated to inputs that have weak impacts on student learning, such as textbooks that are too difficult to comprehend. Second, when the inputs given to schools are substitutes for inputs at home, increasing school inputs may decrease home inputs thus curtailing overall gains in learning (as has been demonstrated in India and Zambia). Third, teachers and other education personnel (who represent almost the totality of education spending) may be poorly motivated to perform. Poorly prepared teachers, high levels of absenteeism, an environment of weak supervision, inadequate regulatory oversight and poorly designed incentives contribute to poor performance.
The 2004 World Development Report Making Services Work for Poor People argued that the underlying cause of such failures in basic service delivery in developing countries are weak accountability relationships between the state, service providers, and the citizens and clients they serve. In the education sector, efforts to strengthen these accountability relationships through system reforms have been numerous. However, designs have varied considerably and there has been little rigorous evaluation of impact until recently.
This impact evaluation cluster is promoting rigorous evaluations of accountability-promoting reforms in basic education in two directions, taking directly from Hirschman’s notion of voice and exit. Countries may decide to strengthen their public systems by increasing accountability in public schools; evaluations from this cluster will show the reforms and changes that can produce the largest impact. Alternatively – or in tandem – countries may decide that some parts of educational provision may be left to the non-state sector (with or without financing from the public sector). The phenomenal growth of low-cost private schools offers a unique opportunity to expand the existing cluster activities to examine different models of private school engagement.
II. Description of cluster and ongoing research.
The Basic Education Service Delivery cluster, established through the previous Spanish Impact Evaluation Fund, focused on promoting rigorous evaluations of three of the most common types of accountability-promoting reforms in basic education: school-based management; information provision to empower school stakeholders; and teacher contracting and incentive reforms. As part of this, researchers followed school-based management reforms in Mexico that attempt to empower parents in rural schools and community-managed schools in Nepal; information for accountability projects in Liberia; and teacher contracting and pay for performance projects in Sao Paulo and Pernambuco, Brazil. Within this cluster, researchers also followed projects in India, Kenya and Pakistan. Overall, we are already drawing evidence from 22 rigorous evaluations in 11 countries. This work led to a book, Making Schools Work, which examines how strategies to strengthen accountability relationships in school systems have affected schooling outcomes. As a result the evidence based grew considerably in recent years.
In summary, information for accountability—on its own—is a potentially useful tool for improving learning outcomes but one whose success at leveraging change is sensitive to many factors. For information provision to be effective, service providers must have the ability to change in response to the accountability pressures it can induce. If providers are circumscribed in their ability to take action, it is unsurprising if information has no impact. Information is therefore critically linked to local decision-making authority. The impact of information is also likely to be higher if it is linked to performance rewards or sanctions. School-based management (SBM) can improve learning outcomes, though the results are mixed. It is likely that for school-based management to be effective, it must entail a real transfer of authority (that is, provide school committees and parents with real tasks) and the building of capacity to carry out those tasks. Nevertheless, recent evidence shows modest but significant gains in learning in some contexts.
Evidence on the use of contract teachers suggests large efficiency gains in student learning outcomes, though caution at the sustainability of such an approach is mentioned. On pay-for-performance reforms the most recent and robust developing-country evidence suggests that bonus-pay incentives can improve learning outcomes, at least in the contexts studied most carefully to date. Nevertheless, the cluster concludes that much more effort at linking such reforms is needed. That is, increased use of cross-over designs in encouraged in future research. Moreover, experimenting with the size of rewards would be warranted.
III. Outline for research agenda.
Despite this effort at producing evidence from rigorous evaluations of education interventions the evidence base is still light. For example, a large and important set of evaluations in Sub-Saharan Africa that have not yet generated results are not yet reported on. Future work should be encouraged along at least three dimensions: Replication; combinations with long-term follow-ups; and new innovative models.
(i) Replication: Given a sound theoretical framework—that is, a logical chain for how an intervention might work to effect change—replication is an important tool for deepening our understanding of what works, where, and why. The number and range of evaluations to date do not support a satisfying synthesis. Replicating successful models in different contexts is necessary to quantify the extent to which programs work under varying circumstances. Replicating interventions that have been successful in small-scale settings at a regional or national level is necessary for confidence about what works.
(ii) Combinations with Follow-ups: The research discussed here points to the potential benefits of interventions that combine information, school management, and teacher incentive reforms. There will be a high payoff to research designs that test various combinations and extensions of these approaches. Some studies have solid crossover designs, and these have played an outsized role in advancing our understanding. In addition, the studies that were successfully completed in the past would benefit enormously from long-term follow-ups that will allow researchers to track intervention beneficiaries into their working lives. This will allow us to investigate the medium-term returns in terms of employment, wages, health and family structure.
(iii) Innovative New Models: Within this cluster, we also propose a new agenda on innovative new models of education delivery with a focus on the low-cost private sector. In many low income countries the growth of the private sector has taken up the lion share of new enrollments in basic education. In Pakistan, for instance, the private sector is one of the fastest growing industries and an increasing source of wage jobs in the formal sector for women. Recent research shows that children enrolled in private sector schools are learning more at a lower cost and report lower gender-bias, better knowledge of history and geography and civic values more in concord with the vision of a democratic state. We propose two basic models for further investigation.
Encouraging the use of the private sector with public finance is a model worth investigating. What could be the role of public finance of private schools – for the poor through direct public finance or through vouchers that encourage choice? Although such evaluations are increasing in scope, many basic design questions (such as whether vouchers should allow top-ups) need to be evaluated to further our understanding of what works and how.
But, in the long-run, experience with such public-private partnerships may be disappointing, since the accountability problems that plague public provision in the first place replicate themselves once public subsidies form a large portion of the financing of private systems. For instance, “government-aided” private schools in India perform as poorly as pure public schools; medical schools in India are often run by politicians with very low quality to take advantage of public subsidies. Therefore, we also propose investigating whether there is a direct model of increasing access to finance among private schools in a pure micro-finance environment.
IV. Research questions and outcomes.
The cluster will focus on basic education quality. The focus will be on (i) innovation among low-cost private schools; (ii) broader quality issues including capacity building; and (iii) tailored and refined work on accountability in the public sector.
(i) Low-cost private schools: Within the existing evidence base of accountability models, the question of how low-cost private schools perform is a missing link. Public support to private schools can take different forms, but typically involves direct support to schools in a charter-type model or financing through the demand-side in a voucher-type model. The key questions to address include: How do private schools for the poor perform? What kind of support works best? How do the various accountability programs work in a system of private provision but public finance?
(ii) Quality Issues: Many efforts to develop education systems comprise of significant efforts to reform teacher training systems. However, despite the attention training has received, and the large amount of resources devoted to it, surprisingly little empirical evidence is available. The aim of the cluster is to ascertain how effective teacher training is, by soliciting research proposals that attempt to understand how incentives can help agents to select more effective forms of training. How can training become more demand-based? How can incentives be used to make teachers more effective? What is the role of training in systems that select more motivated teachers? How can the successful teacher incentive programs be combined with teacher training that will sustain and create continuous improvements? What is the interaction between teacher selection and training? What is the relationship between motivated teachers and incentives to continue professional development efforts? Proposals will be solicited that can also deepen our understanding of the relationship between different pedagogical approaches and what teachers are taught. In addition, we would favor proposals that attempt to investigate how teachers perform in the classroom and how this affects student learning.
(iii) Accountability in the public sector: This area will focus on the impact of various forms of information for accountability, school-based management and teacher incentives programs in other contexts. What is the impact of adjusting the rewards in different programs? For example, adjusting the level of grants received by committees in school-based management programs. Also, changing the level of rewards in teacher pay-for-performance programs. What is the impact of combining the various accountability reforms (information, school-based management and teacher incentives) – cross-over designs? What is the impact of combining various forms of accountability programs with private provision of education? For instance, what is the role of parent associations in private schools subsidized by the government? How do pay-for-performance programs and teacher hiring practices perform under private provision? How does greater use of information on schooling outcomes help parents make better decisions in privately provided, publicly financed programs? How does information help administrators improve performance in publicly subsidized, privately-provided schools?
In most cases the appropriate outcomes to consider are student learning gains. However, since we are still working in countries that have to yet to achieve universal enrollment or completion, then indicators of attendance, enrollment and completion will also be considered. In some of the cross-over designs intermediate indicators should also be considered. In programs designed to test the efficacy of alternative service delivery models, such as private schools for the poor, then attention to equity outcomes is very important. Key to equity outcomes is the question of gender differences; thus all proposals will be expected to at least examine female-male differences in implementation and outcomes, along with efforts to examine heterogeneous effects. All studies should demonstrate strong efforts at collecting detailed cost information so that cluster can make informed analyses of the cost-effectiveness of various reforms.
REFERENCES
Bruns, B., D. Filmer, and H. Patrinos. 2012. “Making Schools Work: New Evidence on Accountability Reforms.” Human Development Perspectives. The World Bank.
The World Bank. 2004. “Making Services Work for Poor People”. World Development Report.
2