The Context of Programme Evaluation
Introduction
Welcome to the first unit of Monitoring and Evaluation of Health and Development Programmes.
The module is based on the understanding that you have a fair amount of experience in some aspects of projects or programmes management and that we are building on this foundation.
Neither monitoring nor evaluation can be effectively undertaken unless the programme being evaluated has a sound conceptual framework. Accordingly, one of the key messages we put across is that monitoring and evaluation processes must be conceptualised early in the Programme Management Cycle. We therefore devote a session to the development of a conceptual framework.
At the end of the unit we invite you to add monitoring and evaluation objectives to the evaluation plan you will develop for your first assignment.
We will use material from a programme for primary prevention of Non-Communicable diseases implemented in Cape Town by staff of the School of Public Health in 2002, as an example.
There are five Study Sessions in this unit.
Study Session 1: An Introduction to Monitoring.
Study Session 2: An Introduction to Evaluation.
Study Session 3: Approaches to Evaluation.
Study Session 4: Locating Monitoring and Evaluation in the Programme
Management Cycle.
Study Session 5: Developing a Conceptual Framework for a Programme.
Study Session 6: Programme Goals and Objectives.
Study Session 7: Monitoring and evaluation objectives.
In Sessions 1 and 2, we define the concepts of monitoring and evaluation and debate their different purposes. Session 3 introduces you to the approaches to evaluation. In Session 4, we will take a look at the components of the Programme Management Cycle and the position of monitoring and evaluation within it.
Session 5 focuses on identifying the four principal elements of a programme, and on the development of a conceptual framework for a programme.
In Session 6, we explore the concepts of programme goals and objectives and discuss their significance in the monitoring and evaluation processes.
Session 7 guides you through developing monitoring and evaluation goals.
Here is an overview of the intended outcomes of Unit 1.
Learning outcomes of Unit 1
By the end of this module, you are expected to be able to:§ Understand key concepts relevant to monitoring and evaluation.
§ Make a reasoned case as to why projects should be monitored and evaluated.
§ Explain the importance of being critically reflective of your own evaluation paradigm.
§ Explain the difference between interpretivisim and positivism.
§ Discuss the strength and weaknesses of your own evaluation approach.
§ Locate monitoring and evaluation in the Programme Management Cycle.
§ Develop understanding of the components of a project using a conceptual framework.
§ Develop programme goals and objectives.
§ Develop monitoring and evaluation objectives.
§ Design and motivate for a monitoring system as part of an evaluation plan.
Please be aware that there is a glossary of concepts that are commonly used in monitoring and evaluation, in section 5 of the Module Introduction. Refer to these explanations when you come across an unfamiliar term.
Preparing for your assignment
By the end of this unit, you should be in a position to begin your first assignment and prepare goals and objectives for a programme as well as a conceptual framework. This will provide the basis for a programme evaluation. Try to develop your assignment as you proceed through the module.
Before you start on Session 1, here is an opportunity to reflect on your existing knowledge of monitoring and evaluation.
Review your knowledge of monitoring and evaluation
It is always good academic practice to review what you already know about the topic and to identify where you need to deepen your understanding. Here is a short self-evaluation task, which will enable you to:
§ Reflect upon your previous experience (if any) of monitoring and evaluation;
§ Review your current knowledge and skills relevant to this topic;
§ Identify your learning and development needs.
Answer this questionnaire while previewing the Module Guide and the Reader.
a) My previous experience of evaluation consists of:
b) I have knowledge about the following aspects of monitoring and evaluation:
c) I have skills in the following aspects of evaluation:
d) What I would like to learn (or further develop) on this topic:
You might like to record your expectations for learning under the
topics used above – knowledge, skills and experience.
When you have finished this task, briefly scan through this Module Guide and read the table of contents and reflect on these questions:
e) Does it appear that your expectations will be met?
f) Does the material appear relevant to your needs?
g) How can you use this module to meet your learning and development goals?
h) Make notes about any aspects which are missing and send these to your lecturer.
i) What else can you do about these missing aspects?
Hopefully this task has helped you to get to grips with what lies ahead, what you already know, and what is expected of you by the end of the module.
Resources available to you
There are many areas throughout this module where additional references, readings or websites are given. These are avenues where you can follow up material that is of particular interest to you.
Your lecturer can also help you find additional materials in monitoring and evaluation. Make contact early in your studies so that she may get to know you and your learning needs. A starting point will be to fax your lecturer a copy of your audit task with your name on it. Take a look at it again when you have completed the module to see whether you feel that gaps still exist. Then alert us to them when you evaluate the module.
Unit 1 - Session 1
An Introduction to Monitoring
Introduction
The fact that you have chosen this module suggests that you already recognise the importance of monitoring and evaluation within the life of a programme or project.
We devote this session to clarifying key concepts. We will also raise some of the debates related to the processes of monitoring and evaluation. We emphasise the importance of monitoring and evaluation within the life of a programme, an issue that you, as Programme Manager, may have to take up with your colleagues.
Contents
1 Learning outcomes of this session
2 Readings
3 What is monitoring?
4 Why we monitor programmes
5 Identifying monitoring activities
6 Using District Health Information Systems for monitoring
7 Planning monitoring activities
8 Session summary
9 References
Timing of this session
In this session there are three readings and five tasks. It should take you about two and a half hours to complete.
1 LEARNING OUTCOMES OF THIS SESSION
By the end of this session you should be able to:§ Explain the concept of monitoring.
§ Discuss the role of monitoring.
§ Plan monitoring activities.
2 READINGS
There are three readings for this session. You will be referred to them in the course of the session.
Author/s / Publication detailsMwadime, R. et al. / (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy, & Tufts University: 1.15-1.16.
Feuerstein, M-T. / (1986). Ch 6 - Using Your Evaluation Results. In Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London: Macmillan: 160-165.
Rossi, P. H. & Freeman, H. E. / (1993). Ch 4 - Program Monitoring for Evaluation and Management. In Evaluation: A Systematic Approach. Newbury Park, Ca: Sage Publications: 163-213.
3 WHAT IS MONITORING?
Start this session by considering these examples of monitoring.
§ When we weigh children between 0-6 years at six-week intervals, we are monitoring children’s growth.
§ When you check the patient’s temperature twice a day, you are monitoring the patient’s condition.
§ When you check your bank balance every month, you are monitoring your finances.
Now go through Reading 1; use it to help you to define monitoring in Task 1
READING
Mwadime, R. et al. (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy, & Tufts University: 1.15-1.16.
TASK 1 – DEVELOP YOUR OWN DEFINITION OF MONITORING
What is monitoring? Write your own definition.
FEEDBACK
Now compare your notes with the following definition of monitoring:
Monitoring is a process of continuous and periodic surveillance of the physical implementation of a programme, through timely gathering of systematic information on work schedules, inputs, delivery, targeted outputs, and other variables for the programme, in order to have the desired effects and impact. Monitoring is an integral part of a management support function and of the Management Information System. Monitoring is thus a management tool. Monitoring reports can be used as a basis for internal review (evaluation) of programme operations at the management and technical levels.
Sometimes monitoring is referred to as process evaluation because it concentrates on what is done in the process of running a service or programme, i.e. “within”. This includes the programme activities, the personnel who perform activities and other matters of implementation.
4 WHY WE MONITOR PROGRAMMES
What is the role of monitoring within the life of a programme? Why do we monitor programmes?
In general, the purposes of monitoring are:
§ To oversee the physical implementation of the programme by providing information on its current status, thus ascertaining that implementation is proceeding as planned;
§ To provide the basis for corrective measures to be taken through identifying early indications of deviations, performance gaps and other problems requiring immediate attention for the programme to succeed;
§ To verify proper utilisation of programme resources, including that inputs are made available on time and that inputs are transformed through activities into outputs;
§ To verify that activities are undertaken and transform into outputs.
Note that monitoring activities are carried out around inputs, outputs and processes, that is, activities which enable inputs to be transformed into outputs. Monitoring data should be collected from the inception of the programme, and it should be utilised on an ongoing basis for programme improvement. If there is no monitoring system in place at the inception of the programme, it is not too late to start at a later stage in the life of the programme.
TASK 2 - IDENTIFY REASONS FOR MONITORING A PROGRAMME
Think about a Primary Health Care programme with which you are familiar. List the reasons for conducting monitoring activities in this programme. If there are no monitoring activities, are there any reasons why there should be?
Now compare the reasons you have listed with those in the feedback.
FEEDBACK
Monitoring of programmes is done in order to:
§ Assess the quantity, quality and timing of programme inputs;
§ Verify that inputs are transformed, through activities, into outputs that generate results;
§ Provide information to improve targeting;
§ Identify operational constraints to programme effectiveness thus helping managers to improve implementation;
§ Determine whether a programme is servicing the target groups.
5 IDENTIFYING MONITORING ACTIVITIES
The programme narrative in the box below is based on a programme for primary prevention of non-communicable diseases, which was implemented in Cape Town by staff of the SOPH in 2002.
After reading the narrative, determine which activities of this programme lend themselves to monitoring.
AN INTERVENTION FOR PRIMARY PREVENTION OF NON-COMMUNICABLE DISEASES: EXPERIENCES FROM KHAYELITSHA, A BLACK TOWNSHIP IN CAPE TOWN, SOUTH AFRICA (2002)
Objectives:
To increase the knowledge of Community Health Workers (CHWs) who participate in the programme about 5 risk factors for non-communicable diseases (NCDs) by the end of 2006.
To reduce the prevalence of obesity among CHWs participating in the programme by 10% by the end of December 2006.
To increase the number of CHWs attending all training sessions on primary prevention of NCDS by the end of December 2003.
Methods: Twenty-three CHWs employed by Zanempilo, an NGO, in Site C, Khayelitsha, participated in the programme. Firstly, focus group discussions were conducted with seventeen purposively selected CHWs to explore attitudes, beliefs and perceptions regarding hypertension and diabetes. Secondly, interviews assessed basic knowledge about causes, prevention and control of hypertension and diabetes. Thirdly, anthropometric measurements (weight, height, waist circumference) were taken and Body Mass Index (BMI) was kg/m2 was calculated and used as an indicator of obesity. Fourthly, level of physical activity and associated barriers were assessed. Lastly data was collected on eating patterns (type and frequency of consumption of certain foods). Focus group discussions revealed that CHWs were uncertain about the causes of hypertension and diabetes and found it difficult to grasp the fact that even people without risk factors of these diseases, such as overweight and family history, could be diagnosed as hypertensive/diabetic. Many CHWs believed in traditional medicines and home brewed beer as best treatments for hypertension, and that bitter substances could be used to dilute blood sugar levels in diabetic patients. They also believed that people who took medical treatment became sicker, and their health deteriorated rapidly. A large percentage of CHWs were overweight and some were extremely overweight. Barriers to physical activity included fear of losing weight and lack of environmental safety. Eating patterns revealed a lack of nutrition related knowledge.
A training program for the primary prevention of non-communicable diseases was developed and implemented: The objective of the training was to build the capacity of the Community Health Workers (CHWs) in making healthy choices about the food they eat and in increasing physical activity.
The training programme emphasised the role of nutrition, physical inactivity, body weight, smoking and alcohol in causation of hypertension and diabetes.
Implementation: Training sessions were held once a week for a period of six months. Pre- and post-tests were given to assess the effectiveness of the training. Lectures were given at a central location in the community. Facilitators of training workshops came from different tertiary education institutions and the training was participatory. Learning was reinforced by use of visual aids and real animal organs where applicable. Case studies, specifically developed for this population group, were used to reinforce learning and to encourage critical thinking. For each topic, its role in causation of NCDs was explained, preventive measures were highlighted and a key message was provided which could be used to inform the community.