1

Module 9: Program Evaluation

L. Richard Bradley, Ph.D.

Purpose of this Session: Given the increasing emphasis on accountability and educational outcomes, teachers are being asked - and in many cases, required - to provide data which supports their use of innovative educational strategies such as service-learning. Participants in this session will learn

  • Why, what, when, and how to evaluate; and
  • How to use findings to build school and community support and enhance

future programs.

Getting Started

Put the following questions on newsprint or on a dry-erase board and invite participant responses. Write down responses without comment.

  1. When I hear the word “evaluation,” the first thought that crosses my mind is ….
  1. The best experience I ever had with evaluation was ….
  1. The worst experience I ever had with evaluation was ….
  1. The most pressing question/concern I have about evaluation right now is ….

Clarifying Terms:
  • Research - theoretic frame, scientific design, control for causality, robust analysis, validity/reliability, generalizes
  • Program Evaluation – funder-driven, biased sample, focus on specific program and anticipated outcomes, lack of controls for causality
  • Evaluation Research – coherent program design and outcomes, procedures allow for causal inferences, clear implications beyond idiosyncratic program that was evaluated

TRANSPARENCY - “Clarifying Terms”

Clarifying Terms

  • Research - theoretic framework, scientific design, control for causality, robust analysis, validity/reliability, generalizes
  • Program Evaluation – funder-driven, biased sample, focus on specific program and anticipated outcomes, lack of controls for causality.
  • Evaluation Research – coherent program design and outcomes, procedures allow for causal inferences, clear implications beyond idiosyncratic program that was evaluated.

©, 2005, S. Gelman, A. Furco, B. Holland, and R. Bringle, “Beyond Anecdote: Challenges in Bringing Rigor to Service-Learning Research,” paper presented at the 5th International Service-Learning Research Conference, East Lansing, MI, November 13, 2005.

Introduction

Two assumptions guide this session.

1. Teachers are already “swimming in data;” they just need help in clarifying what is important and what is not; and

2. The goal of good evaluation, at least in the field of service-learning, is not to “prove” something; but rather to better understand why a particular program did or did not work.

Directions for Opening Exercise Using Case Study Approach

  1. Divide participants into teams of 5-8 persons/team and have them sit around tables or in circles.
  2. Distribute and go over handout Steps in the Evaluation Process.
  3. Distribute the handout “Evaluating Your Project: Getting Started.”(see also Service-Learning Quick Sheet # 18: Evaluating Your Project – Getting Started)
  4. Distribute copies of Case Study on Evaluation.
  5. Assign teams to focus on one of the four middle schools described in the case study.
  6. Give each team newsprint and markers. Ask one person in each group to serve as “scribe.”
  7. Read over the Directions for the Case Study.

“You and the members of your team have just been hired to work with the teachers of one of the four schools described in the Case Study to develop an evaluation plan. Using the handouts, “Evaluating Your Project: Getting Started,” and “Steps in the Evaluation Process” to guide your thinking and planning, work as a team to develop a preliminary evaluation plan for service-learning implementation efforts at your school.”

  1. Allow 20-30 minutes for teams to work together to develop preliminary evaluation plans.
  2. Allow 20-30 minutes for teams to present their preliminary evaluation plans.

Note: If workshop participants have already designed a service-learning project using the process outlined in Module 4, or if they already have a project in mind or underway, use their project as the basis for this exercise.

Refer to TRANSPARENCY: STEPS IN THE EVALUATION PROCESS

Distribute HANDOUT

Choosing an appropriate evaluation strategy can be made easier by careful attention to questions relating to:

1)Decide on the focus of your evaluation.

a. What do you want or need to evaluate? (e.g., number and ethnic background of

participants, number of beneficiaries, type of program)

  1. What questions would you like to be able to answer? (e.g., the level of

administrative, colleague, and community support for program)

2)Decide on the purpose of your evaluation.

  1. To satisfy the requirements of funders?
  2. To help make decisions about whether to continue a program?
  3. To guide related educational reform efforts?

3)Decide on the audience with whom the results of your evaluation are to be shared.

a. Is the audience basically “in house” or will the results be shared with the general

public?

b. Why do they need this information?

4)Decide on the resources (dollars, time, people) needed to conduct your evaluation.

  1. What resources do you have?
  2. What resources do you need?

5)Decide on an appropriate timetable for your evaluation.

STEPS IN THE EVALUATION PROCESS

STEP ONE: Decide on the focus of your evaluation.

STEP TWO:Decide on the purpose of your evaluation.

STEP THREE: Decide on the audience with whom the results of

your evaluation are to be shared.

STEP FOUR:Decide on the resources (dollars, time, people)

needed to conduct your evaluation.

STEP FIVE:Decide on an appropriate evaluation timetable.

The decisions you make at each step will help you select the most appropriate evaluation strategy for your program. Strategies include:

qualitative vs. quantitative

formative vs. summative

descriptive vs. experimental

external vs. internal

@, L. Richard Bradley, Ph.D., 1999

Types of Assessment

  • Reflection: Activities producing information oriented toward the self-assessment of persons who are engaged in the experience
  • Process Evaluation: Activities producing information about how a class, course, or program was implemented.
  • Outcome Evaluation: Activities producing information about what outcomes occurred as a result of a class, course, or program.
  • Correlational: Activities producing information about what relationships exist between aspects of a class, course, or program.
  • Experimental Research: Activities producing information about why a specific outcome occurred.

©, 2000, Bringle and Hatcher, Michigan Journal of Service-Learning

If you want to do “research” rather than evaluation, there are some things you need to pay attention to:

Principles of High Quality Research
  • Poses significant questions and uses methods that permit direct investigation of those questions; uses multiple indicators and methods
  • Is linked with and guided by a strong theoretical base; clear line of reasoning
  • Multiple comparable units with large samples; controls for differences among groups
  • Is replicable and generalizable
  • Is scrutinized and critiqued by qualified professionals
  • Quality matters most – less research on outcomes and more on the processes that lead to high quality programs

©, 2005, S. Gelman, A. Furco, B. Holland, R. Bringle, “Beyond Anecdote: Challenges to Brings Rigor to Service-Learning Research,” paper presented at the 5th International Service-Learning Research Conference, East Lansing, MI, November, 2005.

Planning Your Evaluation Program

Using The “Backwards Design” Process

  1. What would you like your service-learning and/or character education program to look like one year from now? What would you like your program to be known for?

List ideas without comment.

  1. How would you know whether you were successful in achieving your goals? What would constitute concrete, visible, measurable, evidence that you had achieved your goals?

List – focus on being concrete – for example, % increase in test scores, attendance, increases in numbers involved, etc.

  1. Develop a plan for getting you from here to there. In other words,
  • What kinds of information will you need?
  • How will you collect this information?
  • How will you analyze this information?
  • When will you collect this information?

(See pp.39-40 for a “checklist” to help you answer questions 2 & 3).

©, L. Richard Bradley, Ph.D., 2004

Establishing S.M.A.R.T Objectives

Short-term objectives translate broad, long-term goals into prescriptions for action. They are specific, measurable, attainable, relevant, and time-bound – S.M.A.R.T.

Specific

  • Not vague or general - wanting “students will be more respectful of their peers” is a worthy goal, but how do you measure it?
  • Objective: incidents of bullying will decrease by 50%.

Measurable

  • Wanting female students to have a greater sense of confidence in science and math is a worthy goal, but how do you measure it?
  • Confident students are more likely to enroll in advanced courses and will probably earn better grades in those courses.
  • Objective: 20% greater female participation and achievement in advanced math and science courses.

Attainable

  • Good objectives are not so far from current performance levels that they cannot be achieved, but they must represent a challenge – a manageable stretch from present levels and serve to inspire rather than discourage effort.
  • Objective: Student performance on standardized tests in reading will increase by 5% rather than by 50%.

Relevant

  • Good objectives are relevant to the core mission of your school.
  • It would be possible to formulate objectives that meet all the other criteria about but are not really relevant to your school’s broader mission.
  • For example, a school team might establish as an objective that 86% of students bring their lunch to school. This is specific and measurable, but who cares? What connection does it have with the core mission of the school?

Time-bound

  • Good objectives have specific deadlines.
  • They are not to be achieved “some time in the future,” but rather “by May of next year” or some such date.
  • A time limit allows people to focus their energies by setting a cut-off date.
  • Deadlines also put issues of accountability in perspective – certain objectives are to be met this year, others next year, and so on.

Charlotte Danielson (2002). Enhancing Student Achievement: A Framework for School Improvement.Alexandria, VA: ASCD, pp.126-127.

Module 9: Case Study on Evaluation

School District A, located in a suburb of a large mid-western city, and School District B, located within the city itself, each received Learn & Serve America grants during the recent school year. Both districts were new to service-learning, although many teachers were known to be involving their students in various kinds of community service activities such as canned food drives and collections of toys during the holiday session.

Most students in District A come from upper middle-class homes. 95% of the students in District A graduate from high school. A high percentage go on to college. Students from District A score at or above “passing” on 25 of the 27 state mandated proficiency tests. Administrators and educators in District A view service-learning as a way to involve students in experiences they might not otherwise have. Particularly important to them is the role of service-learning in reducing stereotypes and prejudice in relation to the world outside their school and community.

Most students in District B come from lower middle-class homes. 44% will drop out of school before graduation. Of those who graduate, only about 50% go on to some form of post-secondary education. Students from District B score at or above “passing” on only 7 of the 27 state mandated proficiency tests. The new Superintendent of District B has made raising these scores, particularly in the areas of reading, citizenship and math, a high priority. Administrators and educators in District B are also concerned about high rates of absenteeism, school violence, and other forms of anti-social behavior. Service-learning is being embraced as a possible strategy for dealing with these issues.

To prepare for the implementation of service-learning, teams of teachers from two middle schools from each district attended the same service-learning training conference. The team from District A included middle school Language Arts, Social Studies, Math, and Science teachers. The team from district B included middle school Language Arts, Social Studies, Science and Family & Consumer Science teachers.

Following the conference, teachers from each district met with colleagues at their respective middle schools to begin planning ways to infuse service-learning into their middle school curriculums. Conversations focused on ways to how to design projects, finding appropriate service sites, the importance of reflection, and ways to “prove” the value of their efforts (evaluation).

District A

District A has only two middle schools. Teachers at both decided to implement service-learning programs.

  • The principal at LongfellowMiddle School was enthusiastic about the opportunities for new learning that might be opened up to students through their service activities. His commitment to successful implementation was demonstrated by giving team members an extra planning period twice a week. His only request was that, at the end of the year, teachers be able to provide him with information about the numbers of students involved, what they did, how many people benefited from their service, and some data on “student impacts.” Teachers at Longfellow spent most of the first semester planning. During the second semester, two service-learning activities were implemented. During the first nine weeks, one group of students studied about poverty and the Great Depression and made regular visits to a homeless shelter. A second group of students learned the math and science skills needed to undertake a stream-monitoring project. During the second nine weeks, the teams switched projects. Reflection was done regularly.
  • The new principal at ElderberryMiddle School was somewhat more cautious in her endorsement of service-learning. While she was thought it was a good idea, she was concerned about it might take away from preparation for proficiency tests. Due an unexpected increase in student enrollment, it was not possible to give teachers time during the regular school day to plan and coordinate service activities. She was less interested in how many students participated in service-learning project, than in demonstrating academic outcomes. Teachers spent all of the first semester planning and then implemented a four-week cross-age math and reading tutoring project with students at a nearby elementary school in the late spring. Reflection was not done regularly.

District B

Two inner-city middle schools in District B were selected by district administrators to participate in implementation efforts. Criteria included principal and teacher willingness to support the program, as well as school challenges such as low proficiency test scores and high rates of absenteeism and disciplinary referrals. Teachers understood that continuation of service-learning programs beyond the first year would be heavily dependent on hard data supporting improved academic performance and decreases in behavioral problems. Teachers also knew that many people in the community did not understand or approve of their efforts to get young people out into the community doing service.

  • The principal at Harrison Park Middle School thought service-learning might be a good idea, but was concerned about the time it might take away from efforts to raise the reading and math skill levels of students at Harrison Park. She and the teachers who would be involved already had a “full plate” and were concerned about adding anything more at this time. She was also not sure that she would be able to allow students from the school to leave school to do their service activities. However, they agreed to give service-learning a try at the request of the Superintendent. Teachers spent all of the first semester planning and were able to come up with a four-week in-school beautification project, which they did, in early April, following proficiency testing. When the project was finished, students had to write a short essay on what they did and what it meant to them.
  • The principal at TrumanMiddle School faced many of the same problems as the principal at HarrisonPark, but was more open to the potential service-learning might hold for the school and its students. He had heard about service-learning efforts at some other schools with similar student populations and thought it might help with both academics and student attitudes. Although it was difficult, and caused other teachers to complain, he found a way to give his implementation team time for planning time once-a-week. He also helped with efforts to find service sites for students. Teachers spent most of the first semester planning. An eight-week service-learning program, involving multiple sites – including sorting food at a local food bank, helping out at a shelter for homeless families, and working with senior citizens on a gardening project – resulted. Students visited their service site once a week, reflecting after each visit.
  1. Move to a summary of material contained in Module 9, highlighting the following information as needed (and as time permits).

WHY EVALUATE?

WHAT TO EVALUATE: TOWARD A NEW MODEL

WHERE TO BEGIN: SOME SUGGESTIONS

INITIAL CONDITIONS FOR SUCCESS

LISTENING TO THE ENVIRONMENT