Projects Progressand Achievements in the year 2011
1NSP 2nd Follow up Impact Evaluation:
1.1Background of NSP Evaluation;
The National Solidarity Program (NSP) is the flagship program of the Government of Afghanistan. Since its inception in 2003, NSP has established 22,500 Community Development Councils across 361 districts in all of Afghanistan’s 34 provinces and has financed over 50,000 projects involving water and sanitation, rural roads, electrification, irrigation, and human capital development.
In recognition of the importance of understanding how this path-breaking program affects Afghan communities, in 2007 the Ministry of Rural Rehabilitation and Development (MRRD) and the World Bank commissioned researchers from Harvard University to undertake an independent impact evaluation of NSP using state-of-the-art methodologies. Working in cooperation with the Vulnerability Analysis Unit, the evaluation team recently completed the first follow-up survey for the evaluation, covering communities in the provinces of Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar that were previously surveyed in the baseline survey between August and September 2007 and a first follow-up survey between May and October 2009.
Based on the Memorandum of Understanding between the Impact Evaluation Team and the Vulnerability Analysis Unit (VAU) in the Ministry of Rural Rehabilitation and Development (MRRD) which has clarified the role of each party and the reporting chain, and thus to facilitate the efficient and accurate collection and dissemination of data during and after the second follow-up survey. The VAU is responsible to support the Evaluation Team in conduction of SNP Impact Evaluations in all aspects of evaluation essentially, to manage the data collection and data processing activities.
The Prospective Evaluation of Phase-II of the National Solidarity Programme (NSP) seeks to rigorously measure the economic, political, and social impacts of the NSP program on participating communities such as; local governance structures, access to services and infrastructure, economic activity, female participation in community activities, political opinion, and social cohesion. A report summarizing the evaluation results will be delivered to the Government of Afghanistan, donor agencies, and implementing NGOs after completion of second follow-up survey which is going to be done at the early year 2012.
1.2Survey Scope:
Based on the survey operation plan 5,000 households, 500 communities, 10 districts (Adreskan, Gulran, Farsi, Chesht-e-sharif, Balkh, Khost-wa-freng, Sang-e-takht, Dulina, Hesarak and Sherzad) and Six provinces (Herat, Balkh, Baghlan, Dykundi, Ghur, Nangarhar) identified for launching of the survey.
1.3Design of the Survey Instrument:
The survey instruments were drafted by The Evaluation Team in English and werearranged for translation in Dari. Six types of questionnaires such as; male household, female household, male focus group, female focus group, village map and radio form were designed and were printed out for data collection aspects based on the survey sample plan. Likewise, other instruments like; survey guideline, survey books, historical cards, literacy cards, occupation cards and etc. were prepared and printed out to facilitate the data collection activities.The pilot test of the draft questionnaires has launchedin 3districts of Kabul province (Estalif, Shakardara and Mirbacha kot).
1.4Hiring and Training of Survey Supervisors, Enumerators, and other Survey Staff:
Around 140 candidate surveyors from various provinces were called for survey trainingin, Kabul. The training was launched for 17 days in three stages and among 120 candidates 65 were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey. The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.
1.5Administration of data collection:
The second follow-up survey data collection launched followed the completion of training, selection of survey personnel and finalization of survey instruments and implementation guidelines.Sixty-five selected surveyors and supervisors divided to 4 teams,each consist of one Supervisor, 3 Editors and 12 Enumerators (in couple bases) in accordance with residential area and familiarity of surveyors to assigned locations. Subsequently, each teamrepeatedly subgrouped to 2 couples (4 Enumerators with one Team Leader) and one Editor in each. The data collection process was started on May 2011 and was completed almost end of October 2011. The data collection activities opened in Adreskan district of Herat province and followed by Balkh, Khost-o-Freng,Gulran, Daikundy, Hesarak, Farsi, Dulina, Chesht-e-Sharif and Shirzad districts in a sequential order. Financial and budgetary arrangements pertaining to the second follow-up survey were well managed and finalized in consultation with the NSP Finance Department, the World Bank, and the Evaluation Team. The budget and expenditures were arranged in accordance with the survey monthly action plan and were documented based on the NSP financial policy and regulations.(For information about project budget and expenditures see Annex1)
NSP 2nd Follow up Survey, Total Household and Focus Groups QuestionnaireLocations / HH Questionnaire / FG Questionnaire / Actual Form Exist
No / Province Name / District / No of District / No of Villages / Male / Female / Total HH / Male / Female / Total HH
1 / Herat / Adreskan / 1 / 50 / 468 / 458 / 926 / 49 / 48 / 97 / 1,023
2 / Baghlan / Khost-o-firing / 1 / 50 / 500 / 500 / 1000 / 50 / 50 / 100 / 1,100
4 / Balkh / Balkh / 1 / 50 / 500 / 499 / 999 / 50 / 50 / 100 / 1,099
5 / Heart / Gulran / 1 / 30 / 277 / 277 / 554 / 30 / 30 / 60 / 614
3 / Nangarhar / Hesarak / 1 / 42 / 420 / 300 / 720 / 42 / 30 / 72 / 792
6 / Daikundy / Sang-e-takht / 1 / 52 / 515 / 515 / 1030 / 41 / 43 / 84 / 1,114
7 / Herat / Farsi / 1 / 50 / 487 / 487 / 974 / 40 / 47 / 87 / 1,061
8 / Ghur / Dulina / 1 / 37 / 359 / 359 / 718 / 35 / 35 / 70 / 788
9 / Nangarhar / Shirzad / 1 / 36 / 360 / 0 / 360 / 36 / 0 / 36 / 396
10 / Heart / Chesht Sharif / 1 / 50 / 468 / 448 / 916 / 42 / 41 / 83 / 999
Grand Total / 10 / 10 / 447 / 4,354 / 3,843 / 8,197 / 415 / 374 / 789 / 8,986
1.6Monitoring and Supervision:
As monitoring and supervision activities are the most important and credible question for reliability and confidentiality of the survey, particularly, during data collection, therefore, the VAU is seriously considering on this topic and rigorously conduct various methods of monitoring during and after completion of data collection.
a)Direct Monitoring: While the data collection of NSP 2nd follow-up survey was in progress,around 12 field monitors were sent to 10 covered districts to check the quality and progress of data collection in aspects of following of “Implementation Guideline” as well as noticing of enumerators and supervisors problems were encountered in the field. All monitors conveyed a standard supervision form outlined for checking of data collection activities and manifestation of deficiencies and gaps undertaken by field staff. They implied their revisory recommendations to found problems defects. Findings were reported to office and were considered for next plans.
b)Track Points by TRACK 24: Modern technology such as; TRACK 24 and GPS devices were used as online monitoring by central office. Each couple of assigned teams conveyed a TRACK 24 which directly was connected to office PCs and was online controlled each 10 minutes with exact information about location of surveyors and data collection progress. GPSs were also used to track point each interviewed household, which will be used for map designing and household determination in future.
c)Re-monitoring and Data Quality Checking: After completion of data collection in each district a monitor was sent to the same district to check and compare the collected data with several local agencies and stakeholders to verify the accuracy of obtained data.
1.7Data Processing Management:
Based on the MoU(Memorandum of Understanding) between MRRD and Impact Evaluation Team the VAU was expected to manage all data processing arrangements of NSP 2nd follow-up survey pursuing the data collection of the survey. Behind the pilot test and finalization of survey questionnaire, a database using the last version of MS Access, a user friendly package, was designed to facilitate the manual data entry of the survey. A team of Data Entry Clerks, supporting data entry process were hired and well trained. Database and data entry process were piloted and finalized for actual data entry occasion. All necessary technology and equipment such as, desktop computers, stabilizers, UPS batteries and etc. required for data processing response to an electronic database were procured and assembled. The completed survey forms were transferred to the data entry center in Kabul at regular base intervals. The data entry process was started after completion of data collection in first district and behind checking of filled forms in central office on 19th June 2011. It was progressively continued up to 31th Oct 2011. The double data entry system was designed for consistency check and quality control, and the process has started awhile after single entry was initiated. The double data entry for 2nd follow-up survey continued without work stoppage till last returned questionnaire and was completed on 30 November 2011. The data entry process rigorously supervised by VAU management and datasets were supplied to Evaluation Team in regular basis of one-week interval. Daily discovered problems in data entry, database structure and database appending process were solved by VAU Data Processing Management Team.
Data Entry & Data Double Checking Progress till end of December 2011No / Province / District / Single Entry / Double Entry / Checking Number (Double System) / Checking Text
(Double System) / Data Checking / Database Restructure / Triple Entry / Checking Number (Double System) / Checking Text
(Double System) / Questionnaire
Data Checking
1 / Herat / Adraskan / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
2 / Baghlan / Khost-o-Firing / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
3 / Herat / Gulran / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
4 / Balkh / Balkh / Done / Done / Done / Not Done / Not Done / Not Done / Not Done / Not Done / Not Done / Not Done
5 / Daikundy / Sang-e-takht / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
6 / Nangarhar / Hesarak / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
7 / Herat / Firsi / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
8 / Ghur / Dulina / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
9 / Nangarhar / Shirzad / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
10 / Herat / Chesht-e-Sharif / Done / Done / Done / Ongoing / Ongoing / Done / Not Done / Not Done / Not Done / Not Done
2Village BenefitDistribution Analysis (VBDA) Survey:
2.1Background:
This pre-analysis plan for the Village Benefit Distribution Analysis (VBDA) specifies methodologies and specifications to be employed in testing hypotheses pertaining to the impact of institutional changes generated by the National Solidarity Programme (NSP) and related interventions on local governance outcomes. The plan was written prior to data analysis and pre-commits the authors to defined specifications for estimating impacts.
The plan is structured around four sections, one table, and three appendices. Section II provides an overview of the study, detailing the nature of the treatment and associated interventions, the sample, and the experiment deployed to generate data for the study. Section III lists the hypotheses for the study. Section IV describes the methodology and specifications to be employed in hypothesis testing. Table I lists the hypotheses and indicators for the study. Appendices I – II list questions included in the Village Benefit Distribution Surveys and Appendix III lists questions from the Second Follow-Up Survey that are to be used in the analysis.
2.2Description of Dependent and Explanatory Variables and Data Collection:
The VBDA is a behavioral experiment designed to identify how externally-imposed institutional changes generated by NSP impact the performance of local leaders in delivering public services. The experiment will also identify impacts of mandating the participation of different groups of local leaders in service delivery.
The experiment generates quantitative measures of leader behavior by organizing and then monitoring village-level distributions of food aid across 500 villages in Afghanistan. The distribution of food aid provided by external agencies is a common public service performed by local leaders in rural Afghanistan. Such distributions present leaders with the option to distribute aid equitably to vulnerable villagers, distribute aid equally among all villagers, or embezzle aid for personal or familial benefit. Data on how leaders distribute food aid are to be collected by surveys administered to random samples of village households and aid recipients following the distribution.
The sample for the study comprises a sub-sample (to be selected based on accessibility given local security and weather conditions) of 500 villages spread evenly across ten districts in Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar provinces.5 The ten districts were selected based on size, security conditions, and the constraint that no villages in the district had previously participated in NSP. In each of these ten districts, NSP Facilitating Partners were asked to select 50 villages which they deemed suitable for NSP implementation,6 which in turn provided the sample of 500 villages.
Although the 500 sample villages cannot - by virtue of their clustering in only ten of 398 districts and the non-random selection of the ten districts in which they are located - provide a random sample of villages in Afghanistan, the 500 selected villages are representative of Afghanistan’s geographic, ethnic, and economic diversity. An assessment of the demographic and economic characteristics of the 500 villages reveal few substantive differences with those of a random sample of villages surveyed by the 2007-08 National Risk and Vulnerability Assessment.
2.3Hiring and Training of Survey Supervisors & Enumerators:
Around 200 candidate surveyors from various provinces were called and trained in different training courses and different regions such as; Balkh, Kabul and Herat provinces for administration of VBDA survey. The training was launched in various sections and stages, among 200 candidates were participated in these trainings around 100 of them were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey. The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.
2.4Structure of Village Benefited Distribution and Administration of Data Collection:
The food aid distribution and the associated data collection generated quantitative data on the behavior of village leaders in the performance of a key public service, which served as the dependent variable for the study. Distribution and data collection activities were implemented during three visits to each village made by distribution agents and enumerators over three, non-consecutive days. The data collection activities was started on 15th July and completed on 31st Oct 2011.
Day 1: A distribution agent was hired and trained by the evaluation team, traveled to the village and convened a short meeting of the village leaders. The agent informed the leaders that the village has been selected to receive a fixed quantity of wheat (approximately one 50 kilogram bag for one sixth of the households in the village) from the World Food Programme (WFP) and that, in three days, WFP will be delivering the wheat to the village. Leaders were informed that the wheat was intended for distribution among vulnerable members of the village community, such as widow-headed households, and were directed to determine which households in the village are to receive the wheat. In accordance with WFP monitoring requirements, leaders were asked to prepare a list of households in the village that will receive wheat, complete with the amount that each household will receive, and were informed that the list will be collected by a WFP representative on the day of delivery.
Day 2: Three days later, a different distribution agent returned to the village with the quantity of wheat was allocated for distribution in that village, which was transported either by a WFP vehicle or the vehicle of an agency contracted by WFP to conduct the delivery. The agent collected the list of wheat recipients were prepared by the village leaders and, after verifying the legibility of the list, authorized the transfer of the wheat to the village leaders.
Day 3: Ten days after the distribution of wheat, a team of 12 male and 12 female enumerators returned to the villages to administer household surveys to three groups of villagers:
First Survey: Random sample of households in the village;
Second Survey: Random sample of households that were listed by village leaders to receive wheat;
Third Survey: Random sample of households that First Survey respondents indicated received wheat, but which were not included in the list provided by the village leaders.
Survey questionnaires are administered conterminously to a male and a female respondent in the same household. Questionnaires were administered to 12 First Survey households and 15 total Second and Third Survey households.18 First, Second, and Third Survey questionnaires were identical and were collected basic demographic and socioeconomic data on the characteristics of the respondent’s household, as well as information on the distribution and characteristics of recipient households. (For information about project budget and expenditures see Annex1)
VBDA Survey Coverage DetailsLocation / # of Questionnaires and Households Interviewed
ID / Province / District / Total Villages assessed / 1st Survey-Male / 1st Survey-Female / 2nd Survey-Male / 2nd Survey-Female / 3rd SurveyMale / 3rd Survey Female / Total-Male Questionnaire / Total-Female Questionnaire / Total HHs Questionnaire
1 / Herat / Adreskan / 43 / 667 / 685 / 329 / 364 / 158 / 169 / 1,154 / 1,218 / 2,372
2 / Balkh / Balkh / 50 / 592 / 593 / 424 / 427 / 246 / 246 / 1,262 / 1,266 / 2,528
3 / Baghlan / Khost-o-Firing / 47 / 669 / 669 / 531 / 531 / 120 / 120 / 1,320 / 1,320 / 2,640
4 / Herat / Farsi / 45 / 632 / 643 / 440 / 445 / 113 / 118 / 1,185 / 1,206 / 2,391
5 / Nangarhar / Hesarak / 38 / 570 / 570 / 378 / 378 / 192 / 192 / 1,140 / 1,140 / 2,280
6 / Herat / Gulran / 28 / 425 / 424 / 236 / 253 / 94 / 102 / 755 / 779 / 1,534
7 / Daikundy / Sang-e-takht / 50 / 706 / 710 / 381 / 388 / 73 / 73 / 1,160 / 1,171 / 2,331
8 / Ghur / Dulina / 26 / 369 / 294 / 298 / 235 / 45 / 35 / 712 / 564 / 1,276
9 / Nangarhar / Shirzad / 36 / 541 / 0 / 480 / 0 / 54 / 0 / 1,075 / 0 / 1,075
10 / Herat / Chesht-e-Sharif / 41 / 544 / 536 / 439 / 437 / 96 / 101 / 1,079 / 1,074 / 2,153
Grand Total / 363 / 5,171 / 4,588 / 3,497 / 3,021 / 1,095 / 1,055 / 9,763 / 8,664 / 18,427
2.5Monitoring and Supervision: