Course Food Security Policies Formulation and Implementation

Course – Food Security Policies – Formulation and Implementation

Lesson 3 – Policy Monitoring and Evaluation

Food Security Information for Action
Food Security Policy
Formulation and Implementation
Lesson 3
Policy Monitoring and Evaluation
Learners’ Notes

Course – Food Security Policies – Formulation and Implementation

Lesson 3 – Policy Monitoring and Evaluation

Learning objectives

Introduction

M & E in the cycle of policy formulation and implementation

Purpose of FSP Monitoring and Evaluation

Monitoring and Evaluation concepts

Approaches and Methods

Evaluation criteria

Setting up an FSP Monitoring and Evaluation system

Linking M& E and FS Information to other Policy Spheres

Summary

If you want to know more

Annex 1: Case study - M & E concepts applied to a food security policy and related projects and programmes

Annex 2:Example of a basic LogFrame Matrix for a Food Security Policy

Annex 3: Common Food Security Indicators and possible Data Sources

Annex 4: Options and criteria for setting up a monitoring and evaluation unit at central policy level

Learning objectives

At the end of this lesson you will be able to:

•define the purpose of a monitoring and evaluation system for Food Security Policies; and

•identify the approaches and methods to be applied in monitoring and evaluating the implementation and impacts of Food Security Policies.

Introduction

Monitoring and Evaluation (M & E) is an essential element of a policy cycle, i.e. of the process of policy formulation and implementation.

M & E helps to ensure that a policy is effective in achieving its objectives, by reviewing policy implementation and finding out whether and to what extent a policy is implemented as planned and working towards its planned objectives to improve the food security situation.

This lesson presents the concepts and approaches applied in establishing and operating an M & E system for Food Security Policies (FSP).

The food security situation is not only influenced by the implementation of specific food security policies but also by:

other policies (e.g. macro-economic and sector policies); and
changing macro-economic,political and social conditions (e.g. currency exchange rates, world market prices of export and food commodities,civil unrest, war).

Therefore, those factors with implications for food security need to be taken into account in policy M & E.

The M & E results are communicated to the policy makers and other stakeholders, so as to trigger, if required, necessary adjustments in policy design or implementation.

Difference between monitoring and evaluation

Monitoring refers to a continuous observation of the process of policy implementation and of the progress achieved.
Evaluation is a specific and in-depth review done after certain time intervals, when substantive steps of policy implementation have already been made and tangible changes and policy impacts are expected to have materialized and can be assessed.

M & E in the cycle of policy formulation and implementation

The following graph presents an overview of the role of Monitoring and Evaluation in the cycle of policy formulation and implementation:

Purpose of FSP Monitoring and Evaluation

Monitoring and Evaluation of food security policies serves to inform government and other stakeholders of the state of implementation of the policies and the progress towards achieving the intended food security objectives.

M & E provides answers to the following questions:

Are the policy measures implemented as planned?
Is the implementation of the policies bringing about the desired changes and working towards attaining the planned food security objectives/results?
If there are delays in implementation and/or divergence between the planned policy objectives and the actual results achieved, what are the reasons for that divergence? Possible reasons to be considered: Flaws in policy design or implementation, impacts of other policies on the food security situation or changing external conditions which affect policy implementation or results.
What needs to be done to remedy the deficiencies (adjustments in policy design and/or implementation, integration of food security considerations into other policies and programmes)?

Monitoring and Evaluation concepts

A comprehensive system for monitoring and evaluating food security policies incorporates elements of different monitoring and evaluation concepts, as described below.

Table 1: Different Monitoring and Evaluation Concepts

M & E of policies / Policies are set at aggregate levels (macro, national, sectoral) and aim at wide-spread impacts (country-wide).
The subjects of policy M & E are the policy objectives, strategies for and means of implementation and the policy impacts.
A policy M & E system is organized at central level, compiling and complementing the results of programme and project M & E.
M & E of programmes & projects / Programmes and projects are set at lower aggregate levels and are more narrowly defined in terms of objectives set, issues and/or target groups addressed and/or geographic areas covered.
Monitoring at P & P level is organised by the P & P management. Ideally, P & Ps form part of a strategy for policy implementation, and the results of P & P M & E feed into the policy M & E system.
Monitoring of the implementation process / Monitoring of implementation of policies / programmes / projects covers:

financial and other inputs used (accounting / input monitoring);
activities performed (activity monitoring);
outputs achieved (output monitoring);

It compares actual accomplishment with plan.
M & E of results and impacts / It traces and assesses the results of policy / programme / project interventions in comparison with the ex ante situation before the intervention (baseline) and in view of the overall objectives set and possible unforeseen side effects.
Monitoring / It is a continuous exercise, internally organized by the organization in charge of policy / programme / project management, to generate real-time data and information on policy / programme / project performance regarding implementation and results as basis for management control, for being able to trace possible diversions between reality and plan in time and to make necessary adjustments as early as possible.
Evaluation / Specific reviews of policies, programmes, or projects.
A singular or sequential exercise, often externally initiated and organised by high level decision makers, supervising bodies and financing agencies, in order to provide evidence on the overall performance of projects / programmes / policies in terms of relevance, efficiency, effectiveness, sustainability.
Evaluations typically serve as basis for decisions on extension, expansion, modification and/or replication of projects and programmes respectively necessary adjustments of policies.
See Annex 1: Case study - M & E concepts applied to a food security policy and related projects and programmes

Approaches and Methods

Next we will describe different approaches and methods which can be applied in monitoring the implementation and impacts of Food Security Policies. They refer to:

Logical Framework analysis;
Food Security Indicators; and
Data and data sources.

Logical Framework analysis

Logical Framework (Logframe) analysis is widely applied in project and programme planning, management, monitoring and evaluation.

It is also a suitable method for policy analysis, monitoring and evaluation because it helps to:

set out clearly defined objectives for the different levels of policy implementation (project / programme and policy level) against which progress in implementation and actual achievements can be monitored and evaluated;
select relevant indicators for measuring progress and achievements;
reveal the sources of data to be used for measuring the indicators; and
identify conditions which are important for the achievement of the policy objectives. Though these conditions cannot be influenced by the policy, they need to be monitored and responded to, if they change (e.g. world market prices).

Ideally, a Logframe analysis is done at the early planning stage of a policy, programme or project, in order to ensure that implementation is guided by a clear and consistent framework of objectives, and to allow for the establishment of a baseline situation against which the changes induced can be monitored and evaluated.

A Logframe analysis culminates in the establishment of a Logframe matrix.

Levels of objectives / Indicators / Data sources / Conditions
Overall Policy Objective
Specific Objectives
Results
Policy Measures

The Logframe matrix provides a clear and comprehensive framework of the hierarchy of:

interlinked and consistent objectives (1st column);
related indicators for measuring objective achievements (2nd column);
data sources of the indicators (3rd column); and
important assumptions and conditions (4th column).

See Annex 2: Example of a basic LogFrame Matrix for a Food Security Policy

Food Security Indicators

Indicators are measurements of change which serve as sign posts, milestones and benchmarks to measure progress in policy implementation and of the results achieved so far.

Through the use of suitable indicators it is possible to find out:

whether policy implementation is progressing as planned (process monitoring);
which changes policy implementation has brought about already (actual esults/outcomes achieved compared with baseline); and
whether it is on track towards reaching the planned objectives (actual results/outcomes achieved compared with objectives).

Process monitoring indicators
Typical indicators for process monitoring are:

the number and type of food security initiatives launched;
funds and other resources mobilized for food security interventions;
food security expenditures;
geographic areas covered (geographic targeting);
vulnerable population groups and number of people reached (social targeting);
food security concerns considered in other programmes and policies.

See Annex 3: Common Food Security Indicators and possible Data Sources

For the sake of clarity of M & E results, and for reasons of cost- and time-efficiency in collecting the necessary data and analysing them, the selection of indicators should be confined to one or a few indicators that are most suitable for tracking the changes of the phenomena to be observed.

Good and suitable indicators are Sensitive, Measurable, Attainable, Relevant, Time-bound and trackable, summarized as “SMART”, as described in the table below.

Let’s read the table considering, as an example, that the policy objective is to increase food supplies by increased domestic food production. Indicators will be: Food production and Import figures:

Table 2: “SMART” Quality Criteria for Indicators

Criteria / Meaning / Example
Sensitive / Specific to the issue that is intended to be changed, and sensitive to the changes induced. / Both indicators measure the two factors which contribute to food supplies and are sensitive to changes induced by the policy (food production and supplies expected to increase, imports to stagnate or go down).
Measurable / Measurable, objective and unambiguous, not easily manipulated. / Indicators are easily measurable, derived from agricultural production and trade statistics.
Attainable / Attainable by the policy measures and applicable to measure progress towards achieving objectives. / Increase of food supplies is attainable by increased production brought about by policy measures to promote domestic food production.
Relevant / Measuring factors which are relevant, i.e. directly related to the policy objectives to be achieved. / The indicators are relevant, because food production is - apart from food imports - the major source of food supplies
Time-bound and trackable / Varying over time, reflecting at what point in time changes can be expected to happen; trackable on the basis of available data. / Indicators vary over time and can be easily tracked, since the required data are already and frequently available from agricultural and trade statistics.

The application of the “SMART” criteria gives preference to quantitative indicators that are measurable and, therefore, objectively verifiable.

However, sometimes it is necessary to also consider qualitative indicators, if, for example, meaningful quantitative data are not (yet) available, when participatory approaches to monitoring and evaluation (key stakeholder and target group participation) are applied, or for cross-checking (triangulation) of the results of a quantitative analysis.

Examples of quantitative and qualitative indicators

Quantitative indicators: income, growth rates, production figures, land use data, food consumption and expenditure data, prices, marketing volume, imports, food stocks, nutrition status based on anthropometric measurements, etc.
Qualitative indicators: perception of households of their economic status or food situation (better/ worse/same as before); observations, experience and/or expectations of stakeholders and target group regarding changes induced by a policy, programme or project, etc.

In choosing the most suitable indicators, the following words of Albert Einstein should be kept in mind: "Not everything that counts can be counted, and not everything that can be counted counts."

Data and data sources

The selection of indicators and their use in monitoring and evaluation is closely linked to the issues of data availability and quality.

Only indicators for which valid data are readily available or that can be generated on time and in a cost-efficient way should be selected.

The data quality largely determines the validity of the monitoring and evaluation findings. Criteria for data quality are Accuracy, Preciseness and Timeliness (“APT”), as presented below.

Table 3: “APT” Quality Criteria for Data

Criteria / Meaning / Examples
Accurate / Data matching, as much as possible, the actual values or properties of the phenomena being studied. / Food production figures, prevailing market prices, number of malnourished children, number of people below the poverty line.
Precise / Data reflecting the exact (and not just approximate, broad, vague) values or properties of the phenomena being studied. / Exact (instead of approximate) production costs, exact market prices (instead of wide price ranges), specific income groups (instead of households with wide income ranges considered in one group)
Timely / Data on current situation are available on time. / Data for relevant reference periods (baseline, certain periods, most current) are available on time.

Data collection can be a laborious, cost-intensive and time-consuming exercise.

Reasons of cost- and time-efficiency call for a maximum use of data that are available through statistics and/or are the results of relevant studies and surveys done by different organizations (e.g. research institutions, ministries and government departments, development organizations, NGOs).

Once the indicators are defined and clarification is achieved on the data required, the existing information and data sources should be reviewed, so as to see which meaningful data are available and routinely collected.

If the required data are not available, or cannot be obtained in appropriate quality and time, the following options for data generation are to be considered:
Table 4: Options for data generation

Options / Description / Example
Piggybacking on / upgrading of existing data collection systems / Existing statistical services and data collecting systems may not generate exactly the type of data required but similar kind of data. By complementing or upgrading ongoing data collection exercises, the type and quality of data generated can possibly be adapted to the monitoring and evaluation requirements. / Data on household food expenditures are routinely collected through economic household surveys, but not differentiated by different income groups. If a differentiation of food expenditures by income group is introduced in the survey, this will provide evidence on the effect of food security policies on food consumption of low income households.
Conduct own particular surveys for data collection
. / If the data required on particularly important food security issues cannot be obtained from existing data sources, there might be the need to conduct own specific surveys / An assessment of impacts of policies on different vulnerable population groups (e.g. rural poor, urban poor, jobless, small farmers, HIV/AIDS affected households) may require special surveys to be conducted.
Selection of alternative / proxy indicators / If there is a lack of suitable and current data on the subject to be studied, and if a special survey to collect such data is not feasible, alternative indirect or proxy indicators may have to be used on which data are available or can easily be generated. Although not directly an expression of the phenomena to be observed, proxy indicators are related to such phenomena and change in the same direction, thus allowing conclusions on the main subject of concern. / A direct indicator to measure the impact of policies on food security would be a sustainable improvement of nutritional status of a population. If respective data are not available, proxy indicators can be used, such as the nutritional status of children under five in selected sample areas; prevalence of nutrition related diseases; the frequency of appeals for food assistance; the phenomenon of 'hunger migration'; the number of meals consumed per day, etc.

Evaluation criteria

To assess the overall performance of development interventions, a set of evaluation criteria has been developed[1]

which is widely applied in project and programme evaluations, and can analogously be applied to evaluate the overall performance of policies.

The evaluation criteria are the following:

Table 5: Evaluation criteria

Relevance

/ The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.
Effectiveness / The extent to which the policy intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.
Efficiency / A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results.

Impact

/ Positive and negative, primary and secondary long-term effects produced by a policy intervention, directly or indirectly, intended or unintended.

Sustainability

/ The probability of continued long-term benefits of an intervention, even after the intervention as such is being phased out.