Monitoring and evaluation Trainer's Notes

MONITORING AND EVALUATION

Trainer's notes

PURPOSE and content

This module gives participants some general guidelines on how to monitor community drug use interventions, and discusses in more detail how to evaluate. Evaluation of rational drug use interventions directed towards consumers is rarely done systematically, and as a result we have little evidence to prove success and/or failure of various intervention strategies. The module gives several options for evaluation designs, and argues for the need to combine process and effect measurements. It gives an overview of evaluation methods. In the activity participants formulate questions and select methods for the evaluation of a training programme for drug sellers operating in Ugandan communities.

Objectives

Upon completion of the module participants will be able to:

1.  Understand the different functions of monitoring and evaluation.

2.  Define what should be monitored and how this should be done.

3.  Formulate appropriate evaluation questions and identify useful methods.

4.  Make a plan for monitoring and evaluation components of an intervention.

Preparation

  1. Participants should read the Session Notes.
  1. The trainer should find out at least a day in advance of the session which of the participants have been involved in an evaluation of an intervention which aims at enhancing rational drug use by consumers. If there is more than one participant, select the case-study which is most interesting from a training point of view (touches on a variety of evaluation design issues). This participant should be invited to do a 20 minutes presentation on the objectives and design of the evaluation. The presented case study need not be an example of an excellent study. It is in fact better to have a study with flaws as it is easier then to make comments on the design (for example, the evaluation lacks a control group). Explain to the participant though, that the aim is not to criticise his or her design, but rather to learn from the actual practice of doing an evaluation and reflect on how it could have been done better, considering also available resources.
  1. The trainer should read the Session Notes and discuss the presentation with the participant before the session and review the Trainers Notes to see where the case-study can be used as an example or point of reference.
  2. The trainer should check whether there are handouts for the core library and that they are available for distribution after the session. At the end of the session, the trainer should briefly explain why the readings are core.

CORE LIBRARY

Marsh VM et al. (1999) Changing home treatment of childhood fevers by training shop keepers in rural Kenya. Tropical Medicine and International Health, 4(5): 383-389.

Additional reading

Feuerstein, M (1986) Partners in evaluation: evaluating development and community programmes with participants. London, Macmillan.

Fresle DA, Wolfheim C. (1997) Public education in rational drug use: a global survey. Geneva, World Health Organization, (WHO/DAP/97.5).

Loevinsohn BP (1990) Health education interventions in developing countries: a methodological review of published articles. International Journal of Epidemiology, 19(4):788-794.

Øvretveit J. (1998) Evaluating health interventions. Trowbridge, UK, Open University Press.

Piotrow P et al. (1997) Health communication. Lessons from family planning and reproductive health. Westport, CT, Praeger Publishers.

Web sites:

International Conference on Improving Rational Drug Use - Poster presentations

http://www.who.int/medicines/organization/par/icium/prgramme.shtml

INRUD/WHO Promoting Rational Drug Use Course. Decision-Making for Rational Drug Use Interventions module.

http://dcc2.bumc.bu.edu/prdu/HTML_DOCS_TOC.htm

ORGANISATION OF THE SESSION

The session is planned in four parts:

Structure and timing of the session

A. Case presentation - 20 minutes
and questions for clarification - 10 minutes
B. Interactive lecture on monitoring and evaluation - 45 minutes
TEA BREAK - 15 minutes
C. Activity 1 - 75 minutes
D. Feedback and discussion of expert advice from one group, comments from others - 20 minutes

visual aids

  1. Title slide
  2. Objectives session on monitoring and evaluation
  3. Functions of monitoring
  4. Functions of evaluation
  5. What to monitor
  6. How to monitor
  7. What to evaluate?
  8. Preparation
  9. Planning
  10. Implementation
  11. Effect
  12. Four evaluation designs
  13. Four evaluation designs fig. no. 1
  14. Four evaluation designs fig. no. 2
  15. Define key outcome measures
  16. Problems in proving effects
  17. Methods
  18. Examples of key outcome measures
  19. Summary guidelines
  20. An evaluation plan
  21. What can be achieved?
  22. Activity 1


Trainer first briefly introduces the session’s objectives (OH 2)

A. Case Presentation (30 MINUTES)

The trainer clarifies the function of the case-presentation: to relate design issues to a concrete example conducted by one of the participants. It is a learning exercise. The aim is not to criticise the design.

The selected participant gives a detailed presentation on the objectives, methods and results of an evaluation study (20 minutes).

Participants can ask questions for clarification about the presentation (10 minutes).

B. Interactive lecture on Monitoring and Evaluation

(45 MINUTES)

The interactive lecture deals with the following points:

1. Functions of monitoring and evaluation

The trainer gives a brief introduction on monitoring and evaluation dealing with: the differences in functions of monitoring and evaluation (OH 3 and 4).

2. What and how to monitor

The lecture continues with a brief introduction on what and how to monitor (OH 5 and 6). Refer the participants to more details in the Session Notes and explain that in the lecture and activity we focus on evaluation.

3. What to evaluate by intervention phase

Present the various stages of an intervention:

§  Preparation

§  Planning

§  Implementation

§  Assessment of effect

Ask participants to give suggestions for key evaluation questions for each phase. Refer also to the questions in the case-study presented in the beginning of the session. Use OH 7-11.

4. Evaluation designs

Discuss possible evaluation designs. Ask participants to put forward designs – refer to the one presented in the case-study. What else could have been used as design?

Present OH 12-14

Stress the need to measure process variables and effect. Without insight in the process of the evaluation one cannot explain why there was an effect or not.

5. Outcome measures

Discuss key outcome measures and stress that these are based on the communication objectives of an intervention. This means that communication objectives have to be very clear! Give concrete examples of outcome measures. Again refer if relevant to the case-presentation. Use OH 15 and 16.

6. Methods

Ask participants what methods they think are useful for evaluation.

Use OH 17 to review them.

7. Problems in proving effects

Finally, discuss problems in proving effect. Refer to the results of the case-presentation if relevant: use OH 18.

8. Summing up

End the session with the summary guidelines for evaluation (OH 19) and the components of an evaluation plan (OH 20); and a reflection on the time-dimension in effects (OH 21).

BREAK (15 minutes)

C. Activity 1 (75 minutes)

Introduce the activity using (OH 22) and refer participants to the description in the module, see below. This activity tends to be valued by participants as it puts them in the role of ‘experts’. Address them as such when introducing the exercise. These give weight to the activity.

Activity 1

Monitoring and evaluation

This activity is done in ‘table’ groups. You have 75 minutes to brainstorm on the questions and to formulate your advice. The responses should be written down and submitted to the facilitator (maximum of three pages). One group then presents its plan and the other groups comment on and add to this.

Your group has been invited by the funding agency DANIDA to give expert advice on how to evaluate a shopkeeper intervention in Uganda to be carried out by the Uganda Red Cross (URC). The URC has recently received a substantial grant from DANIDA to conduct the intervention, but has failed to include an evaluation component in its project proposal. DANIDA wants you to formulate the evaluation component. The evaluation is to be carried out by a small team of outside experts and national communications specialists in collaboration with the URC, which is implementing the intervention. A detailed description of the rationale for the intervention, its communication objectives, and approach is given below.

Specifically your team is asked to formulate the following:

1.  Evaluation questions (process and effect).

2.  A limited set of key outcome measures, including their purpose, how they should be calculated and what methods should be used to collect data for them.

3.  Advice on whether to use a quasi-experimental design, or whether to limit the evaluation to a time-series design or pre-post design.

4.  A limited set of data-collection methods to be used.

Description of the intervention that is to be evaluated

A previous investigation of drug use patterns in Uganda revealed that the majority (72%) of drug treatment is through self-medication, in that it is undertaken without the advice of a trained health worker. The most commonly used drugs, in order of frequency, are chloroquine, aspirin, paracetamol, septrin (cotrimoxazole) and mebendazole. Only about 25% of drugs are obtained from the Government and NGO units receiving essential drug kits through the Uganda Essential Drugs Management Programme. This is true not only of common analgesics like aspirin, but also of antimalarials and antibiotics, which are obtained mostly from drug shops and general provision stores.

Drug sellers in small shops and storefront clinics were found to be the main providers of drugs for the common conditions of fever, cough, diarrhoea and worms, accounting for around 60% of treatments (Odoi et al., 1996). In addition to drugs, they also provide advice. Medicines sold in the shops are not labelled. While some of these ‘informal’ providers have considerable medical experience and may even be supplementing their meagre incomes in government health units, others have no formal training (Odoi et al., 1996).

In two communities in Tororo District, the URC aims to implement a pilot shopkeeper intervention focusing on the appropriate treatment of fever, the most common condition for which drugs are bought at community drug shops. Malaria is endemic in Tororo. Most fevers are considered to be malaria by patients and self-medicated with antimalarials. The malaria control programme in Uganda encourages the use of chloroquine in adequate dosages as first line treatment.

All drug sellers operating in the intervention communities will be invited to attend a six-day training workshop; they will receive two supportive supervision visits from the trainers; and they will be invited to attend a refresher session.

The training approach is to be participatory and problem oriented, with an emphasis on experience-based learning.

Key messages will be:

§  Chloroquine is the first choice of drug for suspected malaria.

§  Correct dosages for chloroquine for seven different age groups; these correct dosages will be listed on a chart and given to all drug sellers participating in the workshop.

§  It is important to label drugs; for chloroquine sales the drug sellers are given stamps which depict the appropriate dosage of the drug

§  The drug sellers are trained to explain to their customers what the stamped chloroquine labels mean.

§  The drug sellers are trained to ask clients to repeat instruction on dosage of the drugs.

If successful, the URC aims to encourage the Ministry of Health's Malaria Control Programme to implement a nationwide training programme for drug sellers, to increase the rational use of antimalarials by consumers. Malaria treatment guidelines change over time, due to emerging resistance. The URC believes that the most efficient way of informing the general public of changing guidelines is through the drug sellers from whom they usually buy medicines for malaria treatment.

D. FEEDBACK AND DISUSSION OF EXPERT ADVICE (20 MINUTES)

Identify the group which has developed the most elaborate advice and has finished the exercise in the given timeframe. Ask this group to present its results (10 minutes). Others can add their views and give alternative recommendations.

Introduce and hand out any core library materials.

4