The Interventions Planned Under the Project Include

STRENGTHENING AGRICULTURAL PRODUCTION SYSTEMS AND
FACILITATING ACCESS TO MARKETS-
NIGERIA COMMERCIAL AGRICULTURE DEVELOPMENT PROJECT:
IMPACT EVALUATION CONCEPT NOTE

Draft: June 23, 2009

This note outlines the focus and implementation strategy for an impact evaluation of the Commercial Agriculture Development Project (CADP) in Nigeria. The CADP aims at strengthening agricultural production systems and supporting the dissemination and adoption of new technologies, for targeted value chains among small and medium scale commercial farmers in five participating states in the country. These value chains are based on the comparative advantage of each state and include support for staple crop production systems.

This is the first project of its kind in Nigeria and reflects the new emphasis on agricultural growth and diversification of the economy into the non-oil sectors. Public interventions to accelerate agricultural growth have hitherto targeted small and subsistence farmers. The CADP represents an important attempt to make Nigeria’s growth sustainable, increase employment and reduce poverty in rural areas, and to boost investment in new technologies. The project is comprehensive and ambitious in scope and involves significant investments aimed at increasing domestic food production and improving access to markets through the construction and rehabilitation of rural infrastructure.

The impact evaluation will inform project design and test the efficacy of various project components in achieving project objectives. It will provide an estimate of the efficacy of matching grants as a tool to promote the adoption of new technologies, and assess the impact of improved access to markets and information. Finally, the evaluation will be a critical tool to inform scale-up when the program is extended nationwide at the end of the project’s lifetime.

The interventions planned under the project include:

i)Capacity Building (US$4.9M): Training of Commodity Interest Groups (CIGs) and Commercial Agriculture Development Associations (CADAs) to develop their capacities to plan and execute their projects.

ii)Market Facilitation (US$7.9M): Market development, awareness and knowledge sharing for commercial farmers through the creation of a market information system, including market price surveys, website and information kiosks, and quality control measures and standards.

iii)Technology Demonstration and Adoption for Agricultural Commercialization (US$21.6M): Demonstration and dissemination of technology packages in the selected value chains.

iv)Support to Staple Crops Production Systems(US$50.00M): Input support systems, increasing the area under cultivation in existing land holdings, animal traction and power tillers, on-farm primary processing through the provision of threshing/milling machines and storage bins, and post-harvest handling centres.

v)Matching grants(US$15.00 M): One-time capital grant to finance activities aimed at improving the adoption of existing agricultural technologies by participating commercial farmers (iii) and to finance some of the activities to support staple crop production systems (iv).

vi)Development of a Network of Farm Access Roads(US$50.0M): Construction, rehabilitation and maintenance of about 500 kilometres of farm access roads and drainage structures per state.

vii)Provision of Rural Energy(US$30.0M): Rehabilitation and maintenance of rural energy, including the provision of transformers and extension of lines from main transmission line to commercial farmers.

The impact evaluation willaddress four sets of policy questions:

What is the impact of infrastructure interventions including roads and electricity access on market access, proportion of output sold, output processing and profitability?
What is the impact of access to market information on marketed output and profitability? What types of information work best in this regard?
What is the effect of CIG-level support interventions—access to improved seeds for staple crop farmers and technology demonstration and dissemination for commercial value chains?
What is the impact of the different levels of financial incentives in the form of matchinggrants on technology adoption rates among eligible commercial farmers?
What is the effect of the matching grant on production, yields, sales, and marketed output of staple crops, and are there complementary effects on household socio-economic outcomes such as health and education?

The evaluation design will consist of the following three levels:

Level 1: Evaluation of the Rural Infrastructure Component (vi-vii)

Based on a listing of potential areas for intervention in rural roads and energy, locations will be randomly assigned into treatment in three or four phases. The initial listing will involve the identification of more locations than can be covered by the project. Combined with the randomized phase-in, this assignment strategy will allow us to identify short, medium and longer-term impacts of the infrastructure interventions.

Level 2: Evaluation ofthe Agricultureand Market Information Component (i-iii )

This level will be layered on top of level 1. The project will implement the capacity building component (i) in all Commodity Interest Groups (CIGs). The evaluation will test the efficacy of all the remaining subcomponents that are not financed through matching grants through a strategy of randomized assignment of CIGs to different treatment and control groups.

The following table reports the list of all possible combinations resulting from the three sub components

Treatment / Dissemination and demonstration of technology: Commercial value chains only / Support to Staples-Improved seeds: Staple value chains only / Market Information (Information kiosks) / Capacity Building
1. Full treatment / X / X / X / X
2. / X / X / X
3. / X / X
4. Full control / X

Level 3: Evaluation ofthe Matching Grantcomponent (iv)

Matching grants will be used in commercial crop value chains to promote the adoption of known and approved technologies, and in staple crop value chains to finance on-farm primary processing facilities, expansion of area under cultivation and animal traction and power tillers. For staple crops, matching grants involve a uniform contribution of 50% of the cost of the proposal. On the other hand, for commercial value chains, the proportion of beneficiary contribution varies by the cost of the proposal as follows:

Cost of Activities (=N=) / Beneficiary Contribution (%) / Matching Grant (%)
Up to 250,000 / 50 / 50
250,001-500,000 / 60 / 40
500,001-1,000,000 / 70 / 30
1,000,001-5,000,000 / 80 / 20

The evaluation strategy will use a regression-discontinuity design that will build on features of the selection process for the approval of proposals for financing to create a valid comparison group to assess the impact of the matching grant as well as evaluate the cost-effectiveness of different levels of matching grants.

The process for selection of proposals for funding will be implemented as follows. Commodity Interest Groups (CIGs) invite proposals from amongst their members and check that proposals satisfy a simple checklist. The CIGs pass on all satisfactory proposals to the Commercial Agriculture Development Associations (CADAs) for screening and to determine completeness of the documentation. The CADAs aggregate the proposals received from all their subsidiary CIGs and rank these according to quality and priority for funding using a scorecard. The CADAs determine the total number of proposals that can be funded in the year for each value chain, N, and submit the top-N ranked proposals for approval to the State Commercial Agriculture Development Office (SCADO). Theranking of the proposals will vary by value chain, cost category of the proposal and states. The evaluation strategy will rely on the comparison of treated CIGs/CADAs and control CIGs/CADAs around the cut-off point.

This criterion for the selection of projects for funding will allow comparing proposals that received funding because they were ranked just high enough with proposals that were denied funding because they were ranked just below the cutoff. The evaluation strategy will use a quasi-experimental design which compares proposals with rankings just below and above N, which will be on average of the same quality.

Annex 1: Steps in the evaluation design and Sampling Frame

Identify all feasible locations where roads and electricity lines will be constructed over the next 5 years. [Participating States]
Define the road/electricity line precisely in terms of its end-points and all points that the road will go through and important entities it will connect (e.g. market unions). If the road has multiple segments (e.g. AB and BC in the picture above), each segment has to be defined. If one component involves a feeder connection to specific (set of) farms, that should clearly identified. For electricity lines, we need to identify capacity, as well as points at which beneficiaries will be allowed to tap into the lines. [From engineering survey, based on Mushfiq and Nandini’s template, to be conducted by states with support from a field coordinator and the Ministry]
Identify the catchment area for each road and electricity line that encompasses all likely beneficiaries of the infrastructure to be provided. This is the red dotted line in the picture above. The areas defined by the red dotted lines will be our sampling frame. All the data we collect for the evaluation will be from these areas [From engineering survey, based on Mushfiq and Nandini’s template, to be conducted by states with support from a field coordinator and the Ministry]
Identify all (a) market unions, (b) CIGs (c) farmers within each CIG that fall within our sampling frame. [Participating States]
Conduct a listing (i.e. rapid census) of all farms and households that fall within our sampling frame. This will involve visiting each household very quickly to ask only a few questions, including what farms they have and where. We will also need to list farms that are not owned by households within the sampling frame area. We may need to redefine the spatial boundaries of the sampling frame slightly based on this exercise. We will also ask households whether they have applied or plan to apply for a matching grant, and which CIG they are a member of. This visit should take about 15 minutes per household. [Task for the surveying firm or agency, with Technical Assistance from Erich, Nandini, Mushfiq]
With the information collected from the rapid census, we will determine our sample households. We will use a stratified random sampling strategy where we over-sample CIG members and matching grant applicants, but retain sampling weights in order to conduct population-representative analysis for all other questions. [Task for Erich and Mushfiq]
Develop questionnaires and conduct the baseline survey [Task: survey firm, with support and supervision from the states and the Ministry]
Develop questionnaires and conduct surveys of market unions, CIGs, and villages [task: Survey firm with support from the Ministry and states ]
Conduct interventions. Each of the 4 interventions will be assigned in the following way. More detail on the specific nature of each intervention will have to be decided on the basis of templates.
Roads and electricity lines will be phased in. Approximately 25%-30% of planned roads and electricity lines will be built in Jan-April 2010. There are two types of roads: “new construction” (R-NC) and “rehabilitation” (R-R). We will randomize within road type, so that 25-30% of R-NC are phased in during 2010, and 25-30% of R-R are also phased in during 2010. [Task: collaborative – States, Ministry, Nandini, Mushfiq]
Information kiosks will be placed in a random subset of market unions that are within the sampling frame areas (stars in the diagram). We have to offer very structured information content that is consistent across all locations (i.e. same amount of information provided in the same manner). Also, we should provide narrowly and clearly defined information through the kiosks, in order to aid interpretation of the results at the end of the evaluation. [Task: collaborative – States, Ministry, Nandini, Erich, Mushfiq]
Matching Grants – We will use a discontinuity design to compare farmers within our sample whose proposals are successful against farmers whose proposals are unsuccessful.[1] This is the main reason we need to over-sample matching grant applicants using the rapid census. We will need to retain in digital form the entire score-sheets for all applicants. Data on each component of the scorecard will be used later for analysis. [Task – States, Nandini, Erich]
Support interventions (technology dissemination or demonstration and access to improved seeds) will be provided to a random subset of CIGs that fall within our sampling frame areas (diamonds in the diagram). Within each value chain, the technology has to be provided in a structured and consistent manner across locations. Again, we have to narrowly define the technology and the nature of the intervention, so that we can cleanly interpret results at the end of the evaluation. [Task: States, Ministry, Nandini, Erich, Mushfiq]
Conduct follow-up survey around July-August 2010. The exact same set of households surveyed during the baseline (in 2009) will be surveyed again in 2010. Also conduct follow-up surveys of market unions, CIGs and villages at the same time.
Repeat the process (a second round of interventions designed on the basis of year 1 results and another follow-up survey in 2011/12).

Annex II: Scorecard for Matching Grants

Proposed Scorecard for ranking proposals for matching grants: Part I

Supplemental Information: Beneficiaries must submit information along with their proposal

Name of value chain (Please select one):

Oil Palm
Cocoa
Fruit Trees
Rice
Maize
Aquaculture
Poultry
Dairy

Current value of financial and non-financial assets:

Cash in bank (Attach most recent bank statement):

Year / 2007/08 / 2008/09 / 2009/10
Revenues
Costs

Please fill out the following table:

Total cost of proposed technology:

Expected revenues if matching grant is approved:

Current production (Please specify units):

Current yield:

Expected production if matching grant is approved:

Expected yield if matching grant is approved:

Yes
No

Does the proposed technology require electricity?

If yes, then:

Current average daily hours of power supply

Yes
No

Electricity primarily used for farm activities?

Yes
No

Does the proposed technology require water?

If yes, then:

Yes
No

Is the current supply of water adequate water supply?

Yes
No

Is this water primarily used for farm activities?

Please include a brief description of public goods element of proposed technology (Will the new technology benefit others? What proportion of the community will benefit?)

Simple Investment Plan:

Procurement Plan:

Proposed Scorecard for ranking proposals for matching grants: Part II

Facilitators must complete the following information:

Year / 2007/08 / 2008/09 / 2009/10
Profits (Revenues -Costs)

If the proposal is approved and the technology is adopted, please report/verify the following information:

Cost of the proposed technology:

Matching Grant % contributed by the project based on cost of proposed technology:

Beneficiary’s share of cost of new technology

Expected Profits= Expected revenues- (Costs in 2009/10 + Beneficiary’s share of cost of new technology) =

Expected net returns from new technology = (Expected Revenues- Revenues in 2009/10) / (Total cost of Proposed technology)

Expected net returns to beneficiary from new technology = (Expected Revenues- Revenues in 2009/10) / (Beneficiary’s share of cost of Proposed technology)

Proposed Scorecard for ranking proposals for matching grants: Part III

Yes
No

CIG Checklist

Is the applicant the only beneficiary?

If no:

Assessment of number of additional beneficiaries

Proportion of community benefiting

Yes
No

Is the proposal supported by more than 50% of the members in the CIG?

Yes
No

Is the proposed technology an approved technology?

Yes
No

Is there any environmental risk associated with the proposed technology?

If yes: provide brief description

Yes
No

Is the technology appropriate in terms of access to water and electricity?

Yes
No

Is the technology profitable for the beneficiary?

Yes
No

Is the technology profitable in terms of total costs?

Yes
No

Is the beneficiary financially capable of providing matching funds for the proposed technology?

Is the proposed technology in line with CIG priorities?

Yes
No

Do you approve the investment plan?

Yes
No

Do you approve the procurement plan?

Yes
No
Yes
No

Do you approve this proposal?

Proposed Scorecard for ranking proposals for matching grants: Part IV

CADA scorecard:

On a scale of one to ten, please rank the proposal on the following elements:

Alignment with the State Development Plan
Alignment with priorities for the development of the value chain
Alignment with priorities of the beneficiary’s CIG
Economic viability of the proposal in terms of beneficiary profitability
Economic viability of the proposal in terms of net returns to the project from the proposed technology
Financial ability of the beneficiary
Probability of success
Appropriateness of technology given current scale of operations, access to energy, water and value chain
Public goods element and scope for direct and indirect benefits to other beneficiary
Environmental and other risks associated with the proposal

Total score:

Matching Grant share for the project:

Rank within Matching Grant share slab:

[1] Please note that many farmers outside of our sample (both living within the sampling frame areas and others living outside those areas) will apply for and receive grants. However, those farmers will not be a part of the evaluation simply because they will not be surveyed, and we will therefore not have enough information on them. In general, our evaluation is governed by the areas where we are collecting data (the red dotted lines around each planned infrastructure project), and all market unions, CIGs, villages and a random subset of households that fall within those areas. In general, interventions b,c,d can also occur outside those areas, but it’s just that we will not collect data outside these well-defined areas.