Getting Our Story Straight – The Why of Evaluation (Session 1)
Four Stages of Evaluation
There are many ways of thinking about evaluation, but they basically all share a similar process:
1. Design the evaluation
2. Gather data
3. Make meaning from the data
4. Shape practice based on the meaning that comes from the data.
Utilizing Your “So That” Chain for planning and evaluation
Looking from another perspective
You have looked at the “so that” chains by starting from the activity that you conceived, planning and moving toward the goals agreed upon by the ECI/LC—the end that you have in mind. Today you will look at your chain from the end and work backward.
Look at the end and work backward to do two things:
1. Be clear on what the end looks like
2. Put the work in the “so that” chain in the bigger context
1. Be clear on what the end looks like
Begin by talking about what the measures are of the end condition. What are some of the indicators that signal that you have made progress toward the three “big goals” identified in the ECI Theory of Change?
2. Put the work in the “so that” chain in the bigger context]
Look at what else is going on in the situation that is also contributing to the end results.
Looking at context, environment
This process will help you to think about the context/environment in which you are working and to see:
· who you need to partner with for greater impact
· where there might be competing goals
· who is working in parallel with you – even if they are not in a partner relationship
· where there are common bonds
Looking at the link between outcomes and intervention
Consider what measures you would use to assess the outcomes of the interventions. Understanding the context helps determine how to design your evaluation so you can understand the link between outcomes and interventions. In your design, consider what is part of the intervention and what is background/context. You may want to expand your thinking of the intervention to more than what you are doing.
In working through these issues, you are clarifying:
· what the use of the evaluation is
· what the story is that you will tell
· to whom you will tell the story.
Now you can identify the key questions you need to think through in order to
· develop a good evaluation
· effectively measure your outcomes
· understand the link between your desired outcomes and your intervention
Here are some questions to consider during this process:
· What are the key outcomes you want to measure, how, and for what group of children and families?
· By working backward and looking more broadly at what affects the desired outcomes, do you need to make any adjustments in who you work with to achieve outcomes?
· How should you define the intervention that you will look at in regard to telling the story of the link of intervention to outcomes in our particular work?
· What is the story you want to be able to tell?
These are major design questions. They will shape how you will ultimately design your evaluation.
Reflecting and Summing it Up
After doing this exercise, reflect on questions such as, What did you learn about:
· how you make choices of who to work with to achieve desired outcomes
· how much you can/should focus on attribution of outcomes to your particular work
· figuring out the story you want to be able to tell.
Living The Story Together – ECI Cluster Groups (Session 2)
Each state will do some work on its own “so that” chain before working as a cluster. The purpose of this time is to:
· continue the type of work you had done in the previous session, but focused on your own state’s “so that” chain
· reflect and apply the learning from the first session to your own state; and
· continue to focus on evaluation design
Individual State Work
Each state will look at its own “so that” chain and other information and identify what is the scope and nature of the story you want to be able to tell about the relationship between the results for children and families and the nature of the intervention. Take what you individually experienced in the previous session and talk about how it applies to your state’s “so that” chain. As a state, ask yourselves parallel questions to the three discussed earlier in the example “so that” chain and situation.
The questions:
· What are the key outcomes we want to measure, how, and for what group of children and families?
· By working backward and looking more broadly at what affects our desired outcomes, do we need to make any adjustments in who we work with to achieve outcomes?
· How should we define the intervention that we will look at to tell the story of the link of intervention to outcomes in our particular work?
· What is the story we want to be able to tell?
In looking at this, it is important to recognize that the relationship between the results and the intervention within a given context over time is the core feature of most evaluations – and the one to focus on here. With this orientation in mind, move on to summarize the lessons learned about:
· your end goal for children and families
· how change occurs
· collaboration and how you make choices of who to work with to achieve outcomes
· how much you can/should focus on attribution of outcomes to you particular work
· sustainability of movement toward your goal for children and families
· figuring out the story you want to be able to tell
Summarize these points on a flip chart page.
Cluster Group Work
Now come together as a cluster. Compare what you have with regard to lessons learned in the sessions so far. Again, keep in mind that relationships between the results and the intervention within a given context over time is the core feature of most evaluations – and the one to focus on here.
In the comparison of your lessons learned, try to answer these questions:
· What common patterns do you see in the ways that your strategy is being implemented across states?
· What are the key things that are instrumental in making the strategy work well?
· What key similarities and differences are present?
· What patterns do you see across time and locations?
· What commonalities exist in your “so that” chains?
· What commonalities exist in your contexts/environments and what are the implications of these for your work?
· To what extent can you work as a learning community rather than individual states with similar goals?
· What is possible in working together as a cluster that is not possible in any other way?
Preparing for afternoon report out
Using chart paper, markers, etc., decide what story you want to tell about patterns you see across the work of the sites in your cluster and the value of working as a cluster around your chosen strategy.
Cluster group leaders (or appointed spokesperson) will lead in communicating your “story.” Consider using the three rules of storytelling (purpose – truth – action) to frame your story.
1