Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

I.INTRODUCTION

The Second Education Development Project (EDP II) is currently being implemented by the Lao Ministry of Education with financial support from the International Development Agency of the World Bank. The overall goal of the project is to achieve universal completion of primary education. In achieving this goal, GoL intends to do this by increasing primary school enrollment and completion in the 19 poorest districts of the six poorest provinces in Lao PDR, namely, Attapeu, Houaphanh, Luangnamtha, Oudomxay, Phongsaly and Xekong as project beneficiaries (Figure 1).

EDP II has three components. Component One increases access and completion of primary education in the poorest districts by:

(i)financing community-based contracting for classroom construction;

(ii)giving community grants for schooling program to lower the cost of education to communities; and

(iii)providing in-service teacher training.

Component Two improves the quality of primary education by financing textbooks and teachers' guides, and strengthening the assessment of student learning outcomes. Component Three strengthens capacities for policy analysis and management within the MoE.

The evaluation study focuses on Component One. It is being implemented through a decentralized structure, involving Implementation Units or IUs established within the MoE’s central, provincial and district offices, and representative Village Development Committees in some 400 villages. Each of Component One’s three sub-components has its own IU in the MoE’s central office with counterparts in the province and district levels. Each of the six target provinces has a PES (Provincial Education Service) while the 19 target districts have DEBs (District Education Bureaus) responsible at the relevant level for the three sub-components combined. Villages are selected, within the 19 target districts, to benefit from Component One interventions on the basis of a set of explicit criteria, favoring remote communities and ethnic minority groups. The criteria as stipulated in the PAD have been modified slightly as provided in the Project Implementation Manual. The compliance status is presented in Table 1.

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

Table 1. Comparison of Selection Criteria

PAD / PIM / Modified version / Compliance to criteria
The village has been in existence for at least three years / The village has been in existence for at least three years / All villages have been in existence for more than three years.
The village has an existing school which has been in operation for at least a year which does not have all grades / The village already has a school in operation, or the community has been currently enrolling its children in the school of a neighboring village at least a year / All villages have been operating a school for more than a year.
The number of school-age children of the village families is equal to 36 or more / The number of school-age children of the village-families is about 36 or more / All 20 sample villages have school-age children of about 36 or more.
The village has a Village Development Committee / The village is organized and has a Village Development Committee / Only three of the 10 beneficiary villages have organized VDCs formed in October 2005 but all have existing village committees.
If the existing school is built in temporary materials which means they require annual and bi-annual repair and replacement by the poorest communities; has a roof that leaks, furniture unsuitable for school work, dark surfaces which aggravate already poor day lighting / is built in temporary materials which require annual and biannual repair and replacement inputs; has leaking roofs;
has furniture unsuitable for focused schoolwork;
has dark surfaces that aggravate already poor day-lighting;
has no storage space for materials / All beneficiary schools visited are made of temporary/semi-temporary materials and one comparison with permanent structure.
Too cold for the climate of the mountainous and Northern districts / is too poorly insulated for the climate of the mountainous and northern districts. / All schools have no insulation.
The village is composed of at least 80% ethnic minorities. / All villages except one (Namvang in Viengpouka with more Koui than Hmong Mien) are composed of more than 80 percent ethnic minorities, namely, Lao Tao-Kadai, Hmong Mien, Mon Khmer, and Sino Tibetan.
Village selection will also be guided by the principles of selecting 50% off-road villages and 50% on-road villages, and locating schools a maximum of 30 minutes walk or 1.5 km from the village. / Villages are either off-road or on-road, the farthest school of which from the village is 1.2 km.
The village is not already benefiting, or has not within the last two years benefited, from a donor sponsored school construction program. / All beneficiary villages have not benefited from a donor sponsored school construction program for the last two years except Pouhohn in Nalae (comparison) for zinc and cement in constructing their school.

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

II.THE PROJECT

Objective

The objective of the evaluation is to provide information on the cost-effectiveness of the three interventions (school construction, community grants and teacher upgrading) to feed both into the Project’s mid-term review, and once the Project is completed, into future strategy and policy decisions.

Specifically, the baseline survey aims to determine the status of the project beneficiaries before the community-based interventions are implemented. Data gathered would be very useful in planning what community-based interventions are to be put in place. It will provide EDP II implementers information on the intended beneficiaries, and have a concrete basis on what interventions to employ, the phasing, training, and feasibility of community-based approach towards providing primary access to education.

Evaluation Parameters

The evaluation will assess three dimensions of cost-effectiveness (Figure 2):

Education outcomes. Did project interventions improve primary enrolment and completion and reduced repetition in participating villages compared with comparable villages?

Targeting precision. What extent have project interventions reached the poorest?

Institutional empowerment. Has the project increased capacity and social capital at the community level in ways that are likely to increase the sustainability and effectiveness of project interventions?

Specifically, the following key indicators using the Project Design Matrix for Evaluation were translated into:

Education Outcomes. Measuring education outcomes such as:

primary enrollment rates;

primary completion rates;

promotion rate; and

primary repetition rates

Targeting Precision. Input indicators would include:

gender;

household income;

costs of education to parents;

ethnic group disaggregated data;

construction achievement rate;

disbursement rate;

actual expenditures vs. standard cost allocated;

cost efficiency rate of CBC versus private contracting);

equitable distribution of community grants;

efficiency of school units constructed;

number of families or students receiving the grants;

utilization rate of grants;

qualitative analysis of where the grants were spent by the community; and

economic and cost efficiency of in-service teacher training (i.e. Cost efficiency of in-service training vs. multi-grade practice training; cost efficiency of previous training vs. programmed training of the project).

Institutional Empowerment. This set of indicators refers to variables related to the Village Development Committee’s:

existence of VDC

gender representation in VDC;

ethnic representation;

regularity of meetings;

member contribution;

political cohesion (local authorities support to education)/ linkages/networking

collective effort/importance of unity, moral support inculcation, initiatives in terms of project sustainability, community support and interest in education;

regular promotion of education;

goals and aspirations;

independence

leadership;

sense of entrepreneurship for sustainability or village economic capacity;

sense or pride of ownership of the project; and

support from the Project’s district engineer.

Evaluation Questions

The evaluation will answer the following questions:

  • What is the impact of the package of school and community interventions?
  • What is the relative cost-effectiveness of each of the three interventions in the package, and does the effectiveness of one intervention increase in combination with the others?
  • Is the demand-driven, community-based approach to school construction more effective than supply-driven provision in improving education outcomes?
  • What is the cost-effectiveness of the different types of activities that communities choose to implement under the community grant program, where communities can decide whether to use the grants for quality improvements (e.g. basic supplies to all children, basic supplies for classrooms) and/or subsidies for the poorest children (i.e. demand-side incentives)? and
  • Is there heterogeneity in impacts? i.e. did some groups within the target districts benefit more from the interventions than others?

These questions, however, shall be answered during the mid-term and the terminal evaluation when community-based interventions have been given. The current evaluation bears the baseline report which contains only the benchmark data for the sample case study. These data shall serve as bases for the mid-term and terminal evaluations.

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

III.METHODOLOGY

The study employed both quantitative and qualitative methods to analyze data sets.

Quantitative Evaluation

Design. The study was based on the existing number of provinces, districts and villages identified as of the first week of October 2005. This included 40 out of the proposed 380 beneficiary villages selected by the PMU based on a Phasing-in Approach. These villages come from two provinces in the North: Phongsaly and Luang Namtha. Due to synchronization of PMU and project-phasing schedules as well as cost constraints, the Evaluation Team employed a non-randomized quasi-experimental design instead of a randomized design covering all six provinces and 19 districts as originally proposed. Baker’s Evaluating the Impact of Development Projects on Poverty (World Bank, 2000) considers this design as the next best option.

Sampling Framework. The village is the unit of evaluation analysis. Twenty-five percent of the 40 beneficiary villages in two provinces with five districts selected by the PMU as of the first week of October 2005 served as the treatment group. (Figure 4). The matching method employed comparison group.

Fig. 3. Sampling Framework

The Team selected ideal comparison villages that match the10treatmentgroups from a larger survey through the Provincial Education Service. The study was conducted in Phongsaly and Luang Namtha, two of the project areas in the northern areas. Five districts were visited consisting of 20 sample villages, with 10 beneficiaries and 10 comparison groups (Table 2).

Table 2. List of villages by province, by district, by type

Province / District / Village name and type
Beneficiary / Comparison
Phongsaly / Samphanh / Mokkoklouang / Phieng
Axeu / Mokpha
Gnot Ou / Longthang / Nakham
Voupaokang / Tapexeu
Luang Namtha / Nalae / Phavy / Hatchone
Konelang / Pouhonh
Viengphouka / Namlo / Namloung
Namvang / Phouyeakao
Long / Houaytoumai / Mekongtai
Sompanmai / Samsob

The comparison villages were selected using propensity score matching, in which the comparison group is matched to the treatment group on the basis of a “propensity score” using the Project’s eligibility criteria for beneficiary villages. The criteria had a total of 100. Every criterion was allocated a score as follows (Table 3):

Table 3. Propensity scoring

Selection criteria / Propensity Score
The village has been in existence for at least three years / 10
The village already has a school in operation, or the community has been currently enrolling its children in the school of a neighboring village at least a year / 10
The number of school-age children of the village-families is about 36 or more / 10
The village is organized and has a Village Development Committee / 10
The school is built in temporary materials which require annual and biannual repair and replacement inputs; has leaking roofs;
has furniture unsuitable for focused schoolwork;
has dark surfaces that aggravate already poor day-lighting;
has no storage space for materials / 10
is too poorly insulated for the climate of the mountainous and northern districts. / 10
The village is composed of at least 80% ethnic minorities. / 15
Village selection will also be guided by the principles of selecting 50% off-road villages and 50% on-road villages, and locating schools a maximum of 30 minutes walk or 1.5 km from the village. / 15
The village is not already benefiting, or has not within the last two years benefited, from a donor sponsored school construction program. / 10
Total / 100

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

Table 4 presents the propensity scores of the beneficiary and comparison villages as follows:

Table 4. Propensity scoring of sample villages

Villages / Selection Criteria Propensity Scoring
Beneficiary / Comparison
Existence for at least 3 years (10) / School in operation for at least a year (10) / Number of school age children is 36 or more (5) / Organized and has a VDC (5) / School built of temporary materials (10) / Poorly insulated (5) / 80% ethnic minorities (20) / 50% off-road; 50% on road; 1.5 km from village (20) / Have not benefited from any school grant for the last two years (15) / Total
Mokkoklouang / Phieng / 10/10 / 10/10 / 5/0 / 5/0 / 10/10 / 5/5 / 20/20 / 20/20 / 15/15 / 100/90
Axeu / Mokpha / 10/10 / 10/10 / 0/5 / 5/0 / 5/5 / 5/5 / 20/20 / 20/20 / 15/15 / 90/90
Longthang / Nakham / 10/10 / 10/10 / 5/5 / 0/0 / 10/0 / 5/5 / 20/20 / 20/20 / 15/15 / 95/85
Voupaokang / Tapexeu / 10/10 / 10/10 / 5/5 / 5/0 / 10/5 / 5/5 / 20/20 / 20/20 / 15/15 / 100/95
Phavy / Hatchone / 10/10 / 10/10 / 5/0 / 0/0 / 10/5 / 5/5 / 20/20 / 20/20 / 15/15 / 95/85
Konelang / Pouhonh / 10/10 / 10/10 / 0/0 / 0/0 / 10/5 / 5/5 / 20/20 / 20/20 / 15/10 / 90/80
Namlo / Namloung / 10/10 / 10/10 / 0/0 / 0/5 / 10/5 / 5/5 / 20/20 / 20/20 / 15/15 / 90/90
Namvang / Phouyeakao / 10/10 / 10/10 / 5/0 / 0/5 / 5/5 / 5/5 / 20/20 / 20/20 / 15/15 / 90/90
Houaytoumai / Mekongtai / 10/10 / 10/10 / 5/5 / 0/0 / 5/10 / 5/5 / 20/20 / 20/20 / 15/15 / 90/95
Sompanmai / Samsob / 10/10 / 10/10 / 5/0 / 0/0 / 10/5 / 5/5 / 20/20 / 20/20 / 15/15 / 95/85

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

The propensity scores may not necessarily equal or did not have an equal match with the comparison group since it is quite impossible to get a perfect match given the location and nature of each village. Nevertheless, it would also serve the purpose of determining sustainability over time. As well, since this is not an experiment, unmatched group would suffice.

There were four types of respondents of the study, namely, school heads, teachers, village heads, and parents (Table 5).

Table 5. Distribution of respondents by type

Village / Type of respondents
School head / Teacher / Village head / Parent
Number / Number / Number / Number
Axeu / 1 / 1 / 1 / 3
Hatchone / 1 / 1 / 1 / 5
Houaytoumai / 1 / 1 / 1 / 6
Konelang / 1 / 2 / 1 / 5
Longthang / 1 / 3 / 1 / 3
Mekongtai / 1 / 1 / 1 / 4
Mokkoklouang / 1 / 1 / 1 / 3
Mokpha / 1 / 3 / 1 / 3
Nakham / 1 / 3 / 1 / 3
Namlo / 1 / 1 / 1 / 5
Namloung / 1 / 1 / 1 / 5
Namvang / 1 / 1 / 1 / 4
Phavy / 1 / 1 / 1 / 5
Phieng / 1 / 1 / 1 / 3
Phouyeakao / 1 / 1 / 1 / 5
Pouhonh / 1 / 1 / 1 / 4
Samsob / 1 / 1 / 1 / 5
Sompanmai / 1 / 1 / 1 / 5
Tapexou / 1 / 1 / 1 / 3
Voupaokang / 1 / 2 / 1 / 3
Total / 20 / 28 / 20 / 82

Instrument. For primary data gathering purposes, the Evaluation Team made use of an Interview Schedule for village heads and parents administered by researchers and supervised by Survey Specialists. A self-administered questionnaire was used to collect information from teachers and school heads (Appendix 1)

Secondary Data Gathering. The evaluation relied on existing data sets from the MoE where available and reliable, and generated new data where gaps exist. The general MoE data were provided by the Department of Planning and Cooperation while data on Project performance was made available by the Component One Implementation Units.

Primary Data Gathering. Before the data collection started, a letter of request was sent to the Deputy Director of the Department of Planning and Coordination who facilitated the arrangements with the Provincial Education Service, which coordinated with the District Education Bureaus that coordinated with the village heads. The primary data were gathered by the Evaluation Team along with representatives from the Planning and Coordination Division. The Team included two Domestic Survey Specialists and research assistants. One was responsible for Phongsaly; while the other took care of Luang Namtha. Both Survey Specialists supervised researchers assigned in the districts. On the other hand, the International Consultants coordinated with the Survey Specialists as well as trained counterpart staff from the DPC, PES and DEBs. Key informant interviews along with document analysis were done to validate responses. The following data were collected from four types of respondents from four types of respondents (Appendices 1-6):


Fig. 4. Data Gathering Instruments.

  1. School Head Survey Interview Guide
  2. School and enrollment data
  3. Educational outcomes
  4. Ethnicity
  5. Women participation
  6. Cost of schooling
  7. Classroom construction costs
  8. School maintenance
  9. School level expenditures
  1. Village Head Survey Interview Guide
  2. Respondent’s profile
  3. Village profile
  4. Ethnicity
  5. Women participation
  6. Institutional empowerment
  1. Teachers Survey Questionnaires
  2. Demographic information
  3. Educational background and training
  4. Teaching practices and needs
  1. Parents Survey Interview Guide
  2. Parent’s demographic information
  3. Parent’s contribution to schooling
  4. Parent’s contribution to school construction
  5. Parent’s expenditures for schooling
  6. Parent’s aspiration towards education of their children

During data gathering for the baseline, the international Consultants were divided into two teams: one coordinating data gathering, in Phongsaly; the other one, in Luang Namtha. The two teams visited selected villages accompanied by the DPC and DEBs representatives for procedure validation and photo documentation purposes for the First Phase.

1

Madecor Career Systems, Inc.

Draft Baseline Report

EDP II: Evaluation of Impact of Community-Based Interventions

IV. Status of component I indicators

It should be noted that Component 1 indicators as stipulated in the PAD and the DFA covers the six province-beneficiaries (Table 6). For purposes of this case study, the unit of analysis is the village. Hence, one can only infer on the status of the whole project. The current baseline status using the Project Design Matrix for Evaluation will be used for this purpose.