Beneficiary Results Assessment Survey Report

Beneficiary Results Assessment Survey Report

Conducted in 9 project districts

Government of Nepal

Ministry of Agriculture Development

Agriculture and Food Security project

February, 2016

Technical Assistance: Food and Agriculture Organization of the United Nations

Acronyms

AFSP:Agriculture Food Security Project

BRA Survey: Beneficiaries Result Assessment Survey (output survey)

DIME:Development Initiatives for Impact Evaluation

DPSU: District Program Support Unit

DTOs: District Technical Officers

FAO TA: Food and Agriculture Organization Technical Assistance

FY:Fiscal Year

FYM: Farm yard manure

GAFSP:Global Agriculture and Food Security Program

GM: Grinding Machine

GoN:Government of Nepal

Ha: Hectare

HH: Household

ICS: Improved Cooking Stove,

IWM: Improved Water Mill

M&E: Monitoring and Evaluation

MDD: Minimum Dietary Diversity

MoAD: Ministry of Agricultural Development

N: Total sample size (Number)

PAD: Project Appraisal Document

PMU: Project Management Unit

PNW: Pregnant and Nursing Women

PPS: probability proportional to Size

SD: Standard Deviation

VDC:Village development committee

WB: World Bank

WDDS: Women’s Dietary Diversity Score

Table of Contents

Acronyms

1BACKGROUND

2OBJECTIVE

3METHODOLOGY

4SAMPLE SIZE

5LIMITATION

6FINDINGS

6.1Demography

6.2Key indicators of Result Framework

6.2.1Yield of major crops at household level

6.2.2Seed Replacement Rate of major crops

6.2.3Productivity of Livestock products

6.2.4Nutrition indicators

6.3Findings on other relevant indicators

6.3.1Changes in cropping intensity and cropping pattern before and after small irrigation support

6.3.2Changes in proportion of HH adopting improved crop management practices

6.3.3Production, consumption and sales of vegetables at HH level

6.3.4Proportion of improved and local Livestock breed at HH level (goat, buffalo, cattle & poultry)

6.3.5Changes in production and productivity level of milk and eggs

6.3.6HH adopting improved livestock breeds and management practices

6.3.7Plantation of fodder and forage

6.3.8Minimum dietary diversity (MDD) of women

6.3.9Women involvement on groups

6.3.10Proportion of HHs using women drudgery reduction technologies

6.3.11Project contribution in enhancing livelihoods, increasing supply of meat products and enhancing nutritional status

7Lessons Learnt

Annex-1: Details of sample districts, VDCs and sample size

Annex-2 Survey Questionnaire

Executive Summary

This beneficiary resultsassessment survey (output survey)has been conducted as an internal monitoring of project interventions to assess the immediate and intermediate outputs/results of crop, livestock and nutrition intervention and also to determine the adoption level of improved technologies. Information has been collected from 1093 respondent beneficiary households of nine sample districts and 140 Households of seven districts (controlVDCs of baseline). Sampling technique has been used to select the districts, VDCs and beneficiaries. Details are provided in the methodology part. The field survey task was carried out by the project field staff by themselves during the month of November 2015.

The findings of report are presented in three sub headings namely; demography, findings on key result framework indicators and other relevant indicators.

Analysis on the changes in the result framework Indicators

As suggested by MTR mission, DIME and PMU worked together forthe calculationof Key indicators of Result framework with due consideration to the contribution ofAFSP interventions (treatment effect). Due to the delay in the conduction of midline survey[1], the output survey results have been used as preliminary estimate for measuring the changes in the result framework indicators for MTR purpose.

Firstly, for proper comparison, DIME calculated the baseline value based on the same nine sample districts. Double difference was used to calculate the treatment effect of the project except few cases[2] on which simple before and after difference was used; as there was massive increase/decrease in the control group. The final value for result framework indicators was calculated by adding the baseline value calculated for 19 districts to the treatment effect. The following tables give the synopsis of the changes in the key indicators of result frame work:

Table-I:Average productivity of major Crops (yield in tons/ha)

Crop / Baseline nine districts 2013 (DIME Survey) / Output Survey nineDistricts 2015 (PMU survey) / Double Difference (Treatment Effect) / Baseline for all treatment 19 Districts / Updated Productivity Value for MTR) / Percentage Increase in Yield
External control / Treatment / Difference at baseline / Control / Treatment / Difference at follow up/output
(I) / (II) / (III)=(II)-(I) / (IV) / (V) / (VI)=
(V)-(IV) / (VII)=
(VI)-(III) / (VIII) / (X) = (VII)+(VIII / (XI)= (VII)/(VIII)*100
Wheat / 1.337 / 1.387 / 0.05 / 1.30 / 1.61 / 0.31 / 0.26 / 1.40 / 1.66 / 19 %
Maize / 1.692 / 1.964 / 0.27 / 1.80 / 2.33 / 0.53 / 0.26 / 1.90 / 2.16 / 14%
Paddy / 2.216 / 2.743 / 0.53 / 1.60 / 2.80 / 1.20 / 0.67 / 2.90 / 3.57 / 23%
Potato / 5.846 / 5.502 / (0.34) / 4.70 / 5.60 / 0.90 / 1.24 / 4.80 / 6.04 / 26%

Source: Baseline survey database, DIME, Aug 2014 and Output survey report, PMU, 2016

Treatment effects for the crop indicators have been calculated using a standard double-difference. The treatment effect was calculated using the Output survey and baseline, with data from both surveys restricted to the nine districts of the Output survey. The final value for the results framework was calculated by adding the baseline value (calculated using 19 districts) to the treatment effect. One point of note that although there is a positive estimated treatment effect, the yields for Paddy actually did not increase compared to the baseline. The positive effect came from the fact that production dropped in the control group, likely because of poor rains. However, such effects can still be counted as positive as the project likely prevented losses in yield despite the drought.

Table II:Seed Replacement Rate (in Percentage)

Crop / Baseline Survey nine Districts (DIME) / Output Survey nine Districts, 2015 (PMU, survey) / After minus Before Treatment / After minus Before Control / Double Difference / BL value (treatment) [3]
(X) / Treatment Effect, Final RF number (XI)=
(X)+ (IX)
External control / Treatment / Diff. at baseline / External Control ( / Treatment / Diff at output survey
(I) / (II) / (III)= (II)-(I) / IV) / (V) / (VI) =
(V) - (IV) / (VII)= (V)-(II) / (VIII)=
(IV)-(I) / (IX)=
(VII)-(VIII)
Main Paddy / 1.58 / 1.37 / (0.21) / 7.03 / 12.99 / 5.96 / 11.62 / 5.45 / 6.17 / 7.20 / 13.37*
Wheat / 1.13 / 0.67 / (0.46) / 5.26 / 9.80 / 4.54 / 9.13 / 4.13 / 5.00 / 4.60 / 9.60*
Maize / 2.25 / 9.57 / 7.32 / 13.80 / 12.70 / (1.10) / 3.13 / 11.55 / (8.42) / 14.70 / 17.83**
Potato / 0.00 / 19.83 / 19.83 / 3.86 / 27.27 / 23.41 / 7.44 / 3.86 / 3.58 / 16.00 / 23.44**

Source: Baseline survey database, DIME, Aug 2014 and output survey report, PMU, 2016

Note:* used double difference; **used before and after difference

The situation for seed replacement rate calculation was a bit complicated. First of all, to make the data comparable, PMU and DIMEagreed to make uniformity on the SRR definition, by including seeds received only from the reliable sources (Agro-vet, Government (DADO/NARC), Seed producers' group/cooperatives, purchased from a private company, or received from a NGO/INGO). This made changes in the numbers in the baseline report, as it also included seeds purchased from the local market, so the baseline numbers will need to be updated. Next, there was a complication as there were major differences between the SRR values at baseline between treatment and control,specifically for Maize and Potato. This tells us that the control VDCs were not adequate controls, as the use/availability of seeds in these was fundamentally different from the project area. Therefore a bit of an unorthodox strategy was used as per the recommendation of DIME. The treatment effect was calculated using the double-difference for SRR for Paddy and Wheat, whereas used a simple after-before difference for Maize and Potato. As for the other indicators, the final value was calculated by adding the treatment effect from double difference to the baseline value, where the baseline value was calculated using all 19 districts. (The treatment effect was calculated using just the 9 districts covered in the output survey.)

Table III: Productivity of Livestock Products

Livestock products / Baseline Survey 9 Districts (DIME, 2013) / Output Survey 9 Districts 2015 (PMU, survey) / Treatment Effect / Baseline Values ( treatment, 19 districts) / Final RF Value / Effect %
External control / Treatment / Diff / External Control / Treatment / Diff / Double Diff. Or Before-After
(I) / (II) / (III)= (II)-(I) / (IV) / (V) / (VI)= (V)-(IV) / (VII)=
(VI)-(III) / (VIII) / (IX)=
(VII)+(VIII) / (X)=
(VII)/(VIII)*100
Eggs/hen/year / 21.923 / 14.989 / (6.9) / 27.0 / 37.0 / 10.0 / 16.9 / 20.0 / 36.9 / 84.7%*
Cow Milk, liters/cow/year / 226.457 / 232.819 / 6.4 / 249.0 / 377.0 / 128.0 / 121.6 / 263.4 / 385.0 / 46.2%*
Buffalo Milk, liters/buffalo/year / 503.755 / 617.107 / 113.4 / 627.0 / 687.0 / 60.0 / 69.9** / 640.4 / 710.3 / 10.9%**

* Used double difference ** Used simple before-after difference

Treatment effects for the livestock indicators were bit more complicated. For eggs and cow milk, a standard double difference method was used. However, for Buffalo milk, it was feltthat a double difference was unrealistic due to heavy imbalance in between treatment and control at baseline, as well as a very small sample size (N=30 in control) for buffalo milk at follow-up. Therefore, for buffalo milk, a simple before-after difference was used to calculate the treatment effect. The treatment effect was calculated using the output survey and baseline, with data from both surveys restricted to the 9 districts of the output survey. The final value for the results framework was calculated by adding the baseline value (calculated using 19 districts) to the treatment effect.

Table IV:Improved Dietary Practices (Percentage of children)

Indicators / Baseline Survey, 9 Districts (DIME) / Output Survey, 9 Districts (PMU, preliminary estimate) / Double Diff / Baseline number (from all 19 Districts) / Final Number for Results Framework
Control / Treatment / Diff at baseline / Treatment / Control / Diff at follow up
(I) / (II) / (III)=
(II)-(I) / (IV) / (V) / (VI)= (V)-(iV) / (VII)= (VI)-(III) / (VIII) / (IX)=
(VII)+(VIII)
Women consumed animal proteins / 50.75 / 54.99 / 4.24 / 83 / 62 / 21 / 16.77 / 56.00 / 72.77
Women consumed vegetables & fruits / 54.14 / 52.29 / (1.84) / 85 / 50 / 35 / 36.84 / 57.00 / 93.84
Percentage of children having 3 IYCF practices / 45.28 / 31.94 / (13.34) / 81 / 58 / 23 / 36.34 / 42.00 / 78.34

Source: Baseline survey database, DIME, Aug 2014 and output survey report, PMU, 2016

Treatment effects for the nutrition indicators were calculated using a standard double-difference. The treatment effect was calculated using the follow up (output survey) and baseline, with data from both surveys restricted to the 9 districts of the follow-up survey. The final value for the results framework was calculated by adding the baseline value (calculated using 19 districts) to the treatment effect. There were a couple of key differences between the surveys that are worth noting. Although the results framework says that the IYCF indicator should be calculated for children 6-23 months, in the follow-up survey the exact age of the children was not recorded, so the values are for all children from 0-23 months. Additionally, questions about breastfeeding were not asked, which affects calculations on dairy consumption (which is part of the 3 IYCF practices).This was adjusted by assuming that 100% of children consumed milk products in the past 24hr, which was the estimate from baseline. Deviations from following 3 IYCF practices therefore come from the other two IYCF indicators (meal diversity and meal frequency).

1BACKGROUND

Agriculture and Food Security Project (AFSP) isjointly implemented by Ministry of Agricultural Development (MoAD)and Ministry of Health (MoH) in 19 mid and high hill districts of the Mid-western and Far-western development regions of Nepal[4] with funding support of Global Agriculture and Food Security Program (GAFSP), the World Bank (WB) is the supervising entity. The project has been designed to enhance the food and nutrition security of targeted communities through a holistic set of interventions comprising of technology development, dissemination and nutrition related activities. The project aims to improve the livelihoods of crop and livestock farmers, women engaged in household /kitchen garden production and households with pregnant and nursing women. The project intends to serve nearly 162,500 beneficiary households of 190 VDCs directly over the period of five years (April 2013 – March 2018).The total funding of this project is US$ 58 million of which GAFSP contribution is US $ 46.5 million and contribution of GoNis US$11.5 million.

The project coverage is based on its’VDC coverage plan-i.e, on three phases; in the first phase,coverageof two VDCs in each district (from FY 2013/14), second phase covered additional four VDCs totaling to six VDCs (from FY 2014/15) and third phase additional four VDCswith total coverage of 10 VDCs in each district (from FY 2015/16).

In Project Appraisal Document (PAD), provisions have been made to carry out baseline survey, mid-term survey and end project survey to assess the changes/impact on its’ result framework indicators. DIME has been made responsible for such surveys. DIMEhas accomplished baseline survey. Besides, in line with M&E Strategy of AFSP, the project has conducted Beneficiary results assessment survey (output survey) to determine/monitor the immediate outputs and results of the project interventions (crop, livestock and nutrition) in November 2015.

2OBJECTIVE

The main objective of this survey was to monitor the immediate and intermediate outputs/results of project interventions at beneficiary level in the first phase and second phase VDCs and also aims to determine proportion of households receiving benefits from project interventions along with their perception on the usefulness of the project interventions. In specific, the survey will focus on to:

Assess the immediate outputs/results of project activities on crop, livestock and nutrition at beneficiary level
Determine the adoption levelof technologies delivered through project, and gather feedbacks for improvement
Determine the level of awareness of beneficiaries on the environmental and social safeguard aspects

3METHODOLOGY

For the selection of sample districts, all 19 project districts were divided into two strata, high hill and mid hill districts. At the first stage, 9sample districts were selected (approximately 50% of total project districts).Four districts wereselected from high hill districts and five districts from mid hill districts randomly.

At the second stage, within the selected districts, two VDCswere selected from six VDCs (2-first phase and 4-second phase VDCs), on which the project has already started its activities/interventions. Whilst selecting the VDCs in each district, the sampling frame was designed on such a way that one VDCwas selected from first phase[5]VDCs (one of the twoVDCs) and other one VDCwere selected from the second phase4 VDCs (one of four VDCs). Simple Random Sampling method was used to select thesample VDCs.The total number ofsample beneficiaryHHswas estimated based on simple random sampling techniques with due consideration of non-response error and design effect. The number of sample households in the selectedVDCwasestimated based on probability proportional to Size (PPS) sampling method, the weights being the number of beneficiary household of selected VDCs. Within the selected VDC, the sample beneficiaryHH was selected based on systematic sampling method from the beneficiary list of AFSP (crop and livestock).

Project field staffs were used as enumerators for the collection of information from selected households provided by PMU. A XLS data entry sheet was designed to enter the data collected from the field and XLS pivot table was used to analyze the data. This task was carried out by FAO TA team in close consultation with concerned PMU officials.

Please refer to annex-2 for the list of selected VDCs and number of sample HHsselected in the sampleVDCs. In addition, survey was also carried out in 140 households of the 14 external control VDCs set out in 7 districts by the baseline survey. A set of questionnaire was developed to interview the selected HHs and attached herewith in annex-2.

With regardsto the qualitative questions, the respondents were asked to provide their opinion on four ranks, 1, 2, 3 and 4. The score 1 was being the lowest and 4 being the highest. The estimate average score was calculated and compared with median score (2.5) to know on which direction the parameter stands for. Likewise, the data series was divided into four equal intervals, (Score 1-1.75) as Very Low, (1.75-2.5) as Low, as (2.5-3.25) High and (3.25-4) as very high; from which one can determine interval (level) to which the parameter fits on.

4SAMPLE SIZE

The total number of sample HHswas estimated using the following formula for simple random sampling

Where,

n = sample size

SE=Standard error=should be used from past surveys=α * P(in this case, as past information of particular areanotavailable)

α = level of significance(at95% confidence interval, α will be 5%) = 0.05

P= proportion that any element falls in the sample=50%=0.5

N= population=Total beneficiaries of 6 VDCs of 19 districts on which project intervention started till FY 2014/15 (crop and livestock)=30,569HHs[6]

A total of 1233 HHs were interviewed (1093 project beneficiary households plus 140 non project beneficiary households) in this survey. The sample size was estimatedusing Simple random Sampling Techniques with due consideration to correction factor for non-response error considering rate of response of the population at 90%and change in sample design effect[7](@ 2.5 times of adjusted sample.Out of estimated sample size of 1112 HHs, 1093 HHs was interviewed, as the 19 selected HHs were not present in the village at the time of survey (seasonally migrated to India and Terai districts). The sample HHs for control VDCs were the same HHsthat was set out during baseline survey. The list of external control VDCsand list of sample HHs was provided by DIME.

Administration of survey:

Concept note, survey questionnaire, instruction for enumerators,and sampling design of the Beneficiary Results Assessment Survey were prepared at central level by PMU with technical assistance of FAO TA. The name of sample VDCs, sample size and list of sample beneficiaries for interview was provided by PMU to District Project Support Unit (DPSUs). Regional M&E officers provided orientation and technical guidance to DTOs (Supervisors) on the objective and technical aspects (data collection forms and methodology) of the survey. DPSUs with the help of DTO/coordinator provided orientation to enumerators (Service provider field staff) on questionnaire and data collection method at field.

The (field staff) technicians and project facilitators of SP were used for collection of information at field.

DTOsperformed as the role of survey supervisor in the districts and provided technical monitoring and assistance to enumerator (field staff) for the survey. In addition, RegionalM&E officers also monitoredthe field duringfieldwork for validation, quality assurance and necessary support to field staff. The monitoring plan of each districtwas prepared and submitted to PMU and FAO TA before field work by DPSU/DTOs. Besides, PMU officials and FAO TA professionals also carried out field supervision during the survey.