The Heller School for Social Policy and Management
Brandeis University
Sustainable International Development Graduate Program
Spring 2014, Module I
Monitoring and Evaluation
HS 287f -3 – TBD Thu 2-4:50
Laura Roper, Ph.D.
Office Hours: TBD
University Notices:
1. If you are a student with a documented disability record at Brandeis University and wish to have a reasonable accommodation made for you in this class, please advise me immediately.
2. You are expected to be honest in all of your academic work. The University policy on academic honesty is distributed annually as section 5 of the Rights and Responsibilities handbook. Instances of alleged dishonesty are subject to possible judicial action. Potential sanctions include failure in the course and suspension from the University. If you have any questions about my expectations, please ask.
Academic integrity is central to the mission of educational excellence at Brandeis University. Each student is expected to turn in work completed independently, except when assignments specifically authorize collaborative effort. It is not acceptable to use the words or ideas of another person—be it a world-class philosopher or your roommate—without proper acknowledgement of that source. This means that you must use footnotes and quotation marks to indicate the source of any phrases, sentences, paragraphs or ideas found in published volumes, on the internet, or created by another student. If you are in doubt about the instructions for any assignment in this course, you must ask for clarification.
______
Course Description
The terms monitoring and evaluation are used to describe a wide range of formal and informal data collection and analysis techniques, generally aimed at letting us know whether a project, program, or policy is working and why. The process of evaluation is embedded in and affected by planning, organizational and environmental dynamics which affect attitudes towards evaluation, influence which approaches are most effective, and impact on whether monitoring and evaluation is utilized (“learning for action”) or simply languishes in a file drawer somewhere. Because the world continues to struggle with unacceptable levels of poverty and all the inequities and inequalities this implies, those of us who have committed ourselves to pursuing sustainable development have an obligation to be careful stewards of the natural, financial and human resources at our disposal as members of communities and/or employees or managers of development organizations. One essential tool for ensuring accountability is effective program monitoring and evaluation.
This course introduces students to the field of evaluation, including their purpose, design, methodologies for data collection and analysis, and utilization. We will also explore the context in which evaluations are carried out, frequent challenges and pitfalls in conducting evaluation, and some tricks of the trade drawing on written materials and experiences of the instructor and enrolled students as well. Students should leave the course informed consumers of evaluations, capable of thinking through evaluation design and contracting qualified specialists for the more technical aspects of evaluation, and familiar with how to tap into the wealth of additional resources (many web-based). As is consistent with the underpinnings of the Program on Sustainable International Development, we will focus on issues of sustainability and be particularly mindful of gender dimensions of development programs and evaluation approaches.
Core Competencies
Conceptual: On a conceptual level, students will understand
üThe range and purposes of project and program evaluation
üThe links between planning, implementation and evaluation and how all are affected by implicit or explicit theories of change
üThe types of evaluation
üThe range of evaluation designs and methods
üThe trade-offs amongst cost, validity, learning and participation
Skills
üIdentifying and mapping the theory of change in a given program
üAssessing appropriate approaches for process and outcome evaluations, taking into account resources, time constraints, and stakeholder interests
üMethods for mainstreaming gender into evaluations and/or doing gender-focused evaluation
ü Identifying outcomes & operationalizing complex concepts (e.g. empowerment, social capital) and developing appropriate measures
PFamiliarity with different data collection methods, practice in design
üDesign a simple evaluation, based on a project of your choosing or one from among several options provided in class.
üConveying results so they will be used
Sustainable Development Statement
The issue of sustainability will be interwoven throughout the course. One of the outcomes that program evaluations must consider are sustainability issues. We will also be concerned with developing “sustainable” monitoring and evaluation systems -- systems that generate the right kinds of information for making significantly better decision.
Gender Statement
The vast majority of evaluations, unless specifically designed to look at gender issues, fail to do so. There is an unfortunate tendency to assume that if a community or family unit is better off, everyone in that community or family benefits. Given the numerous and well-documented unintended impacts of many development programs on women, from eliminating their access to subsistence plots when the promotion of cash crops takes more land to disrupting important social rituals and exchanges when such things as wells or labor-saving devices, it is essential that the class learn techniques for building gender considerations into evaluations.
Course Requirements
Attendance in all sessions and prompt arrival to class
Preparation of all readings
Participation in class discussions and small group work
Completion of an in-class quiz, a short writing assignment, and take home final based on the project you’ve selected
Course Grading
Reading the “Checklist” before coming to the first day of class 5%
Participation in discussions and group work – 10%
Written assignments and in-class quizzes - 20%
Demonstrated knowledge of all required readings – 15%
Initiative in identifying other source materials and familiarity with optional readings– 5%
Final Exam – 45%
All readings can be found on LATTE
Additional resources you may want to reference are:
www.unicef.org/evaldatabase
www.worldbank.org (click onto learning on upper right hand side)
www.ids.ac.uk/info/index.html
www.alnap.org (Active Learning Network for Accountability and Performance)
www.isnar.cigar.org/gender/evaluation.htm
www.idrc.org
www.surveynetwork.org
http://web.undp.org/evaluation/handbook/
Please Note: I view this class as a learning collaborative where each of you brings important perspectives and experiences that can contribute to class, even if they may not be directly related to the core competencies. For example, some of you may be wizards at navigating the web, others may have a strong understanding of cultural dimensions that sometimes complicate development work and collaboration, some of you may know something about adult learning with its implications of how to best disseminate and utilize evaluation. These and other things are all assets that will enrich the class.
Class Curriculum and Syllabus
As a basic reference book, I recommend you buy Bamberger, et al., Real Word Evaluation, Sage Publications, 2nd edition, 2011, particularly for those of you interested in pursuing this topic. I did not order it from the publisher as it is about $60 a copy, although you can get a better price through Amazon and other book selling sites. This is not required, but it is the single best book for practitioners on evaluation, for those who are interested. In addition, various UN and bi-lateral and multi-lateral agencies have invested a lot in developing tools for planning, monitoring and evaluation. For example the UNDP has a handbook that provide step-by-step guidance (http://web.undp.org/evaluation/handbook), while the European Commission has prepared one for humanitarian evaluation (http://ec.europa.eu/echo/files/evaluation/2007/humanitarian_guide.pdf)
Week 1 (3/12) – Monitoring and Evaluation - The Basic Terrain
Class will focus on: How monitoring and evaluation fits into the planning cycle; the difference between monitoring and evaluation; what a development practitioner might be expected to monitor/evaluate; different approaches to M&E; and the importance of a theory of change.
On the first day of class we are going to discuss “The Checklist” by Atul Gawande, which is posted on LATTE. This article touches on almost every important lesson I want you to take from the class. YOU MUST READ THIS ARTICLE BEFORE COMING TO CLASS.
We’ll watch the video Community about a women’s empowerment and community microenterprise program in Bangladesh. This video will be the basis for a series of exercises on evaluation design in the rest of the class.
Before class also please read:
Carol Weiss, “Understanding the Program,” in Evaluation, Prentice Hall, Saddle River NJ, 1993.
Grant Craft, “Theory of Change: Using a Theory of Change to Guide Planning and Evaluation,” 2006
Week 2 (3/19) M&E Design Choices: How to be Strategic in Your Use of M&E.
This particular class will introduce you to the importance of matching your evaluation approach to program design and needs; we’ll identify different forms of experimental, quasi-experimental, and non-experimental design.
Before class please read
Bamberger, et al. “The Most Widely Used RealWorld Quantitative Evaluation Design” in Real World
Evaluation, Sage, 2011.
Ian Parker, “The Poverty Lab,” The New Yorker , May 17, 2010.
Slocum, et. al. Power, Process and Participation: Tools for Change. Intermediate Technology
Publications, 2005), Chapters 2 and 3
OECD Glossary of terms, 2010 (esp. DAC criteria of relevance, effectiveness, efficiency, sustainability, and impact; and terminology related to evaluation design – baseline, benchmark, ex-ante vs. ex-post evaluation, etc)
In class, we will begin the process of designing an M&E system for Uttaran. We’ll map out the Uttaran theory of change and then identify what might be some key strategic evaluation questions we might want to ask about and learn from.
Week 3 (3/26)– Basic Data Collection Methods for Strong M&E
àNote: There will be an in-class quiz on key concepts covered the previous 2 weeks + your first reading for this week.
We will begin the discussion on building strong monitoring systems, capitalizing on the type of data an effective organization should be gathering in any event (financial, staff performance, project and program) to create descriptive statistics and look at three common methods for gathering information – surveys; open-ended semi-structured interviews, and focus groups. This will be the basis for an assignment, you’ll be handing in on the on April 16 or 17th.
Before class please read
Go to this site and learn/refresh your memory about the difference between mean, median and mode (this will be on quiz):
http://www.algebralab.org/lessons/lesson.aspx?file=algebra_statmeanmedianmode.xml
Michael Sullivan, Algebra and Trigonometry, “Part 2 Descriptive Statistics,” in Statistics: Informed Decisions Using Data, 4th Edition, 2013.
ASSIGNMENT (Due 4/16 or 4/17: You will be designing to instruments that will help you monitor and evaluate the Uttaran program. One is an intake survey for new borrowers. The other is an open-ended, semi-structured (OESS) interview protocol (more instructions forthcoming). The assignment is due April 16 or 17.
Readings for Assignment
Ellen Taylor-Powell, “Questionnaire Design: Asking questions with purpose.” University of Wisconsin
Found at http://s142412519.onlinehome.us/uw/pdfs/G3658_2.PDF and on LATTE
“Types of Scales Used in Social Science Research” found at http://sociology.about.com/od/Research-Tools/a/Scales.htm pay attention to Likert, Bogardus and Semantic differential scales.
Anita Gibbs, “Focus Groups” on LATTE
Jessica Shortall, “Cost-effective Household Surveys: Key Lessons for Implementing a Household Livelihood Survey on a Budget,” USAID, 2009
ACAPS, “Qualitative and Quantitative Techniques for Humanitarian Needs Assessment,” May 2012
From www.endvawnow.org the extract – “Being Smart about Data Collection” (on LATTE)
In class – you’ll divide into groups and we’ll look at what baseline data you might have wanted to gather about Uttaran, what basic monitoring system you might put in place and the type of data it might generate, and what sort of descriptive statistics you’d want to collect quarterly or biannually. We will also circle back to the strategic questions we identified last week and see if we are addressing them.
Week 4 (4/2) The Power of Mixed Methods (Quantitative and Qualitative)
We will spend a bit more time on descriptive statistics and data visualization, then turn to how participatory methods can be used to understand complex change at the individual, family, and community level.
Please spend some time looking at the following links:
1. On the gender gap on salaries in the United Kingdom: http://www.theguardian.com/news/datablog/interactive/2011/nov/23/pay-annual-survey-hours-earnings-visualised AND http://www.theguardian.com/news/datablog/2009/nov/12/pay-salaries-survey-ashe-ons
2. Explore other examples of data visualization: http://www.tableausoftware.com/public/gallery
3. If you’re really interested in visualization, this demo on a powerful data visualization software:
http://www.tableausoftware.com/public/how-it-works (it sometimes takes awhile to load)
Please read:
Slocum, et. Al, Power, Process and Participation: Tools for Change. Intermediate Technology
Publications, 2005), look at the following methods. Document link found in Week 2.
- focus groups
- gendered resource mapping
- gender analysis activity profile
- social network mapping
- wealth ranking
Watch the video on participatory wealth ranking at https://www.youtube.com/watch?v=njmdvpiA99c (it’s a little slow, but very instructive – we will discuss in class. Watch up to 17:30)
Class will focus on matching method to information need. We will do a class exercise using the Uttaran example and consider your M&E needs, your information sources, and what mix of methods you might use to track progress and understand why or why not the program is on track.
Week 5 (4/16) Identifying the Design, Appropriate Mix of Methods, and Refining Indicators based on your strategic question(s)
Assignment due
Before class, please read:
Internews, “Closing the Loop – Responding to People’s Information Needs from Crisis to Recovery to Development: A case Study of Post-Earthquake Haiti,” January 2010. Read especially the methodology sections.
Types of Sampling: Uses, Methods and Potential Problems
Resource:
Andrew Fisher and James Foreit, “Chapter 8 Sampling,” Designing HIV/AIDS Intervention Studies: An Operations Research Handbook, 2002.
Week 6 (4/23) – Tying it All Together
Focus this week: Doing an evaluation is like implementing a program. This week we will review step by step the process from identifying the need for an evaluation to developing a TOR, to design and testing of protocols, to the financing and logistics of implementing an evaluation, the need for periodic feedback and check in, to data analysis and validation, to disseminating results.
Before class, please read:
Bamberger, et al. excerpt from chapter “Pulling it all together”, pp. 395-402 on LATTE
Chapter 7 Sebstad and Barnes, "Microfinance Evaluation Checklist, " Chapter VII in Guidelines for Micro-
Finance Impact Assessment, p. 51-58.
The take home final will be handed out. Much of it will focus on elaborating your program/project M&E design of Uttaran within different parameters. More will be explained at the time.
1
M&E (Roper) - Page 1 of 6