Leadership Development Evaluation Handbook

First Draft – August 2005

Theory of Change Approach to Leadership Development Evaluation

Manuel Gutiérrez & Tania Tasse Theory of Change Approach to Leadership Development Evaluation

Manuel Gutiérrez and Tania Tasse

OMG Center for Collaborative Learning

Background of Theory of Change Evaluation

The theory of change approach to evaluation gained popularity and wide acceptance in the 1990’s through its innovative use in the evaluation of comprehensive community initiatives. Given the inherent complexity of these initiatives – they are typically neighborhood-based and seek multiple-level outcomes across several programmatic areas – evaluators found serious limitations in traditional evaluation approaches and were forced to come up with other approaches that would be suitable for its application to these initiatives. The work of the Aspen Institute Roundtable on Comprehensive Community Initiatives for Children and Families was extremely important in legitimizing and expanding the understanding of the applicability and limitations of the theory of change evaluation approach (see Connell, Kubisch, Shorr, and Weiss, 1995; Fulbright-Anderson, Kubisch, and Connell, 1998). Notably, the Aspen Roundtable provided a forum for funders, evaluators, and practitioners to refine the approach by sharing tools and lessons learned from conducting theory of change evaluations.

The basic description of a theory of change approach to evaluation was defined by Carol Weiss in her seminal paper, Nothing as practical as good theory, included in the first Aspen Institute Roundtable publication on the evaluation of community change initiatives(Weiss, 1995). Essentially, Weiss proposed that a theory of change approach requires that the designers of an initiative articulate the premises, assumptions, and hypotheses that might explain the how, when, and why of “the processes of change”. As part of this approach, designers and key stakeholders are asked to identify key programmatic elements and to indicate how these interventions might lead to the anticipated short-term, intermediate, and long-term outcomes. This process is commonly referred to as articulating an initiative’s or a program’s theory of change. Once the theory of change is made explicit, then it becomes possible for the evaluator to test the assumptions that underlie the initiative or program and to assess its outcomes.

Distinction between Theory of Change and Logic Models

Logic models have been used in planning and evaluation for a long time, preceding the popularization of theory of change evaluation. However, they have gained increased attention in the evaluation field, as they are essential tools in the articulation of a program’s theory of change. While linked and complementary, these terms are often used interchangeably, leading to unnecessary confusion. In Table 1, we provide our understanding of fundamental distinctions between the terms theory of change and logic models (see Kellogg Foundation, 2003, for an excellent guide on the development and use of logic models).

Table 1. Basic distinctions between theory of change and logic models.

Theory of change / Logic models
Overarching set of premises and hypotheses that seek to explain why and how change occurs / Systematic visual representations of the relationships between program elements and the sequence of events that will lead to change
Higher order explanation; panoramic view / More detailed description; frame by frame view
Provides reasoning to guide program design, planning, implementation, and evaluation / Provide roadmap, based on theory of change, for program design, planning, implementation, and evaluation
Conceptual framework with explanatory power / Evaluation and planning tools for articulating and describing the theory of change
Describes the principal assumptions that will be tested in the evaluation / Describe the anticipated short-term, intermediate, and long-term outcomes that will be assessed in the evaluation

Theory of Change Evaluation and Leadership Development:

A Good Fit

There are many valid approaches for conducting an evaluation of a leadership development program. Some of those approaches are presented by other authors in this Handbook. Often, factors such as the characteristics of the specific program, the nature of the evaluation questions, and the resources designated for the evaluation are likely to influence the type of evaluation that will be selected. We believe that, in most cases, the theory of change approach would be a good fit for evaluating leadership development programs for the following reasons:

Leadership and leadership development are terms with multiple meanings and definitions. Given the various definitions of leadership and the wide array of existing leadership development programs, the theory of change approach focus on articulating premises and assumptions is an effective process for clarifying a program’s view of leadership and how this view shapes program activities.

Leadership development is a complex psychological and social process. Describing the change process for individuals participating in leadership development programs may be as complex and challenging as describing the neighborhood change process in comprehensive community initiatives. For that reason, a theory of change evaluation represents a promising approach for systematically tracking and understanding the personal change process in individuals.

Leadership development programs typically involve multiple components and use various interventions. In this case, the observed complexity derives from the nature and layering of the interventions. Given this programmatic complexity, the theory of change evaluation approach can help ascertain how and to what extent different program components contribute to the attainment of anticipated outcomes.

Leadership development programs typically hold the expectation that individual-level changes will lead to organization-level, system-level, and society-level outcomes. In this context, the theory of change approach provides a framework that articulates the anticipated pathways of change and allows evaluators to gather data in order to test out whether, to what extent, and in what contexts individual-level change leads to broader outcomes.

Pathway Mapping and Pathway Maps

Staff at the OMG Center for Collaborative Learning (OMG) have been conducting theory of change evaluations for over ten years. Over this time, through multiple engagements involving the evaluation of comprehensive community initiatives and single-focus programs, we have used a process called pathway mapping to engage stakeholders and articulate the initiative’s or program’s theory of change. More specifically, pathway mapping is t

he process of specifying a program’s desired outcomes and linking those outcomes to program actions and strategies. The pathway mapping process also requires making program assumptions explicit, challenging them when they appear inconsistent or unclear, and reaching consensus on those program assumptions and sequence of events that describe why and how a program works – the pathways to change.

The pathway mapping process produces pathway maps. Pathway maps are logic models that place emphasis on the surfacing of assumptions linking strategies, activities, and outcomes. We prefer using the term “pathway map” rather than “logic model” because it connects the process (mapping) to the product (map)and emphasizes the concept of “pathways of change” that links to the program’s theory of change. For us, then, a pathway map is a particular type of logic model.

Figure 1 provides an example of a pathway map for a community-based employment program. In this example we present the basic elements of a pathway map: a contextual analysis for the program, the program’s strategic focus, its actions/activities, and its intended outcomes (short-term and long-term improvements). Very importantly, we also include the program’s core assumptions that underlie the connections between the basic elements of the pathway map. In the next section, we provide more detailed examples of a theory of change evaluation approach and pathway mapping.

Insert Figure 1 about here

Pathway maps and leadership programs

Representing the theory of change

When designing evaluations for leadership programs, evaluators are likely to be challenged by broad and ambitious program objectives, which makes it difficult to know exactly what to look for when assessing program success. In addition, staff and stakeholders of the programs may not be totally clear -or in agreement about- how and why their program activities should lead to the objectives, and what may be the signs of progress along the road to success.

To help overcome these challenges, OMG relies on a pathway mapping process as the first step in a theory of change evaluation of a leadership program. As we indicated previously, in pathway mapping, staff and stakeholders of a program make explicit their theories and assumptions about how the program works from start to finish. The final product of this process, a pathway map, is a special type of logic model or flowchart documenting how program activities (inputs) are believed to create results (outcomes) over time.

Over the past few years, OMG has used a theory of change approach to evaluate three established leadership development and recognition programs. These programs are: the Rockefeller Foundation’s Next Generation Leaders (NGL), the Eisenhower Fellowships (EF)’s international exchange programs, and the Ford Foundation’s Leadership for a Changing World (LCW). OMG was hired to conduct impact assessments of these three programs. The LCW and NGL evaluations have included implementation studies, as well.

These three programs are quite diverse in their views on leadership, their goals and intended outcomes, the characteristics of program participants, and the strategies used to achieve their goals (for more information on these programs, go to their websites listed in the References section). Nevertheless, in spite of their diversity, we found that the theory of change approach has provided a flexible framework for evaluating each program (for more information on the NGL and EF evaluations, see Gutiérrez and Stowell, 2004, and Gutiérrez and Tasse, 2005).

Details of a pathway map

The Eisenhower Fellowships’ pathway map shown here includes program inputs, such the nomination and selection processes that precede the fellowship, joint program planning, the actual fellowship experience, and resources provided to alumni. The pathway map also details outcomes that are expected to occur for the individual fellows (for example, enhanced professional knowledge), for their organizations (improvements and/or growth in the organization), and for their communities or societies (new programs, institutions, policies being put in place). Ideal outcomes were included in this pathway map to convey the vision that the program is ultimately striving to contribute to; however, they are formatted differently in the pathway map to indicate that these are ideals and that the program does not necessarily expect to be able to measure progress towards them. The arrows from left to right reflect the order in which the Eisenhower Fellowships believe outcomes develop.

Insert Figure 2 about here

Pathway maps may vary extensively from client to client, not only in terms of content but often in terms of format and layout. For example, while EF chose to depict outcomes according to the level at which they occur (individual-level, organizational-level, and societal-level), many programs prefer to state their outcomes according to the time period over which they are expected to occur, and thus they use the category headings Short-term, Intermediate, and Long-term outcomes. Another common variation is that, in addition to program inputs and outcomes, some pathway maps include a Problem Statement or Contextual Statement describing why the program was first started, under what conditions, and to address what problem/s. For example, Leadership for a Changing World’s pathway map begins with a Contextual Statement that includes the following language: “There is a lack of conviction in the public that leadership exists and that local/community leaders can impact social issues.”

The length and degree of detail shown by a pathway map are other variables to be decided upon. OMG has worked with programs to produce in-depth pathway maps and also to produce simpler ones. To fully describe the conditions that led to a program, the activities and resources that comprise the program, and the assumptions delineating how and why outcomes are thought to occur, requires a significant amount of time. The resulting document is generally several pages long and requires the full attention of a reader to grasp. This type of pathway map is valuable for some purposes, including building consensus around program theory when establishing a new program, and re-thinking program design. Regardless of the length of the document, evaluators working with program staff need to balance the level of detail provided on a pathway map with the level of information required to understand the program. Typically, the inclination is to add as much detail as possible to the pathway map. However, very detailed pathway maps will look cluttered and will be hard to interpret.

Shorter pathway maps (like the Eisenhower Fellowships pathway map) are generally 1-2 pages long and highlight only the key activities and expected outcomes of a program. These pathway maps may be less helpful in presenting why the program exists, or the details and logic behind the program and activities. On the other hand, they are simpler to understand and thus they can be used to illustrate a program’s logic to audiences that are unfamiliar with the program, or to those who do not require all of the details behind it. In general, the content, format, and layout of a pathway map should be presented in whatever style best reflects the program at hand; and the degree of detail should be decided based on why the program is doing the pathway map and how they are likely to use it in the future. Even when pathway maps are presented in a more simple layout, key elements of the theory of change such as program context and assumptions are presented as accompanying text.

No matter what the finished document looks like, if a pathway map effectively captures the logic behind a program, then it can be used as a roadmap to guide the evaluation. It specifies what results evaluators should look for, and it often specifies at what time periods those results should occur. Pathway maps also provide a list of key program activities, which is critical if the evaluation includes an assessment of the program’s implementation. Once the pathway map is completed, the evaluator’s next step is to select methods and design data collection activities.

It is important to note that, in the absence of an experimental design (which is not a realistic design for the evaluation of leadership programs) or a comparison group design (which is difficult to implement), the theory of change evaluation approach provides a reasonable alternative to the limitation of the lack of a counterfactual. If an evaluation can show that program activities were fully implemented and that outcomes developed in the way envisioned by staff and stakeholders and documented on the pathway map, then credibility is built for the program theory. In other words, it becomes easier to believe that the program contributed to outcomes in the way depicted by the pathway map.