Leadership Development Evaluation Handbook
First Draft – August 2005
Strategic uses of evaluation
E. Jane Davidson & Jennifer Martineau Chapter 12: Strategic uses of evaluation
E. Jane Davidson & Jennifer Martineau
In recent years, the use of evaluation in organizations for strategic purposes has joined other uses of evaluation in the spotlight. One of the catalysts for the development of strategic evaluation is that leadership development is becoming, more and more, an intervention used to accomplish strategic objectives. Evaluation of these initiatives must also be targeted toward the strategic purposes of the initiative. This chapter will discuss the strategic use of evaluation of leadership development for illustrating the strategic outcomes, and relationship to the strategy of, a leadership development initiative.
Introduction
Almost every major organizational intervention (leadership development or otherwise) has its origins in a strategic vision (needed or aspired for endpoint). From this vision, strategic interventions such as leadership development initiatives are designed. A common plea from clients we have encountered is the need to link evaluation of these strategic interventions more effectively with the organizational strategy and vision from which they were derived. The purpose of this chapter is to explain why this is important and how it can be done.
All too often, strategies are “operationalized” very quickly, becoming a cluster of programs or interventions that rapidly become disconnected. Each program or intervention has a set of goals derived from (but smaller in scope than) the strategy. If the evaluations of those interventions amount to no more than program evaluations in the traditional sense (i.e., evaluation against these lower-level goals, plus some investigation of unintended consequences), then the likelihood of achieving the original strategic intent is considerably diminished.
Strategic evaluations address this problem in two ways. First, they make sure that intervention outcomes are linked back to the overarching strategic goals. Second, they explicitly consider the other components of the strategy and how they [do or should] work together to bring the organization closer to its strategic vision.
Strategic evaluation, in its ideal form, is fully integrated within business processes. Thus, business decisions are data-driven using information that has been collected specifically to determine the effectiveness of a particular business process or intervention in contributing to the organization’s strategic performance. Further, this information is synthesized so that it is useful at a higher level, e.g., to determine which business processes contribute the most to the strategy, which overlap and create redundancies, and whether the effects of the strategic interventions is greater or less than the sum of its parts.
Strategic evaluation is relatively new, and many organizations are hesitant to dedicate resources to evaluation when “the business process should be effective if we’ve designed it right”. Some organizations enlist the help of an impact assessment specialist, who estimates the probable impact of a strategic intervention before it is implemented. But in many cases, this is not followed up with any serious attempt to see whether these impacts did in fact eventuate. Helping organizations understand the importance and power of incorporating strategic evaluation into business processes is the goal of this chapter. We hope to illustrate this point by demonstrating the outcomes of decisions made based on evaluation data in a particular organization.
As you read this chapter, assume that when we refer to an “organization”, the concepts we are describing and illustrating can apply to many different types of entities. Corporations, non-for-profits, non-profits, educational institutions, non-government organizations, governments, communities, and other types of “structures” may all have strategies that guide the direction of their work. Thus, though we will not provide examples from all of these many different types of “organizations”, we believe that the content of the chapter applies to them.
Key Definitions
Before moving on, it may be useful to step back and define a few key terms. Most organizations have what is referred to as a strategic framework. This consists of (1) an organizational vision, (2) a mission statement, (3) a set of overarching values, (4) a set of strategies, and (5) a set of specific strategic goals and actions.
An organizational vision is a desired end state of either what the organization hopes to become or what its target community will look like when its work is done. The organizational vision is often rather idealized and unachievable. Its purpose is to inspire organizational members by providing a clear view of the overarching goal the organization is striving toward. An organizational mission is a statement of what “business” the organization is in, i.e., what part it plays in striving toward the vision. Organizational values are principles (statements about what is considered good, valuable, or important) that are used to guide decisions and actions. A strategy is one of the [usually four or five] key approaches an organization intends to use in order to achieve its mission and/or vision. Strategic goals and actions are the more specific operationalizations of strategy that guide the month-to-month running of the organization.
Strategic evaluation refers to the determination of the value of one or more business processes or interventions with particular attention to (a) its consistency with organizational values and (b) its contributions to organizational strategy and, ultimately, the achievement of the organization’s mission or vision. Strategic evaluation differs from program or intervention evaluation primarily with respect to its intended users. Findings are designed and timed to be useful not only to those implementing the interventions that form part of a strategy, but to those reviewing and reformulating the overarching strategy itself.
Strategic evaluation’s “value added”
In this section, we explore the ways in which strategic evaluation can add value to the key stakeholders of leadership development initiatives. Evaluation helps to keep the leadership development initiative connected to the organization’s overarching purpose or strategy. In the evaluations we’ve been involved with, some of those outcomes or strategies include:
· creating shareholder wealth
· making work more meaningful for employees
· creating a better product for the customer
· creating culture change
· building stronger communities
· building specific knowledge about where to invest in leadership development initiatives
· building stronger schools & universities
· improving the academic experience for all students, faculty, and staff
· ensuring best use of resources
· strengthening the leadership pool (related to succession planning)
· strengthening organizational capability
The first author recently evaluated a senior leadership development strategy spanning 34 government agencies. Three main interventions (programs) fell under the umbrella of the strategy:
1. A multi-year, tailored executive development program aimed at very senior managers who would be ready to lead (or help lead) a government agency within three to five years
2. A three-week executive leadership course aimed at senior managers
3. A two-year Executive Master of Public Administration degree aimed primarily at mid-level managers wishing to move into senior positions
The primary strategic goal was to strengthen and diversify the pool of senior public service leaders so that there would always be “home-grown talent” available to fill senior roles. One of the great challenges with using strategic goals such as this in an evaluation is that they are quite often long-term outcomes. This particular evaluation was conducted after the strategy had been implemented just 18 months to two years. Thus, one challenge was how to make the evaluation strategic and not just a run-of-mill process evaluation with a few short-term outcomes tacked on.
In this particular case, the evaluand itself was a strategy. This made it doubly important to ensure that the evaluation itself was also strategic in its design and reporting. This was achieved in several ways. The first and most important step was to engage senior stakeholders—i.e., those who would be making strategic-level decisions based on the evaluation. They were interviewed to determine what it was that they needed to know about the strategy. As expected, most of the questions raised were high-level, “big picture” issues, although important details were also sought. In addition, it was important to maintain a close working relationship with those overseeing the strategy to make sure that their questions were being answered along the way at a level that made the answers maximally useful for strategic decision making.
The second step was the development of a set of overarching evaluation questions that were fundamentally strategic in nature. These questions guided the evaluation and helped ensure that all data gathered would somehow feed into answers to those high-level questions. In this case, the questions were:
1. What is the strength of the strategy’s “brand” within the public service? What is its reputation/perceived strength in the eyes of those who might benefit from it?
2. To what extent are the public service’s best and brightest leaders attracted and encouraged into the strategy’s various programs?
3. What is the quality of the various strategy offerings (worthwhile content, effective delivery) and how effectively are participants and departments contributing to and taking advantage of these?
4. What impact does strategy have on participants’ (i) effectiveness as leaders within the public service and (ii) intent and success in pursuing careers as top-level leaders within the public service?
5. What is the value of the strategy and its programs as an investment of time, money, and other resources?
What made the overarching evaluation questions strategic? One aspect was the use of concepts such as “brand” that are high-level and strategic in nature. Another was the inclusion of important macro-level contextual issues, such as the degree to which organizational cultures were enabling departments to get maximal benefit out of the strategy. Finally, a strategic evaluation must inevitably include an “overall value” question to reflect the strategic decisions made about the allocation of scarce resources. It is not simply a question of whether the strategy achieved its objectives or even whether the benefits outweighed the costs. The question is always, “Was this the best possible use of the available resources to achieve needed and valuable outcomes?”
One useful tool for helping make the evaluation strategic was to devise a logic model that showed how the main components of the strategy could be expected to achieve the long-term strategic goals. This allowed us to identify the precursors of the truly strategic outcomes that might reasonably have been expected to have started emerging. The second (and in hindsight, most important) part of the logic model was a set of “assumed inputs” – conditions that would have to be in place in order for the strategy to work. These included organizational cultures that supported leadership development, a diverse pool of talented recruits coming into entry-level positions, and untapped senior leadership potential (including women and minorities) within the current group of senior managers.
Another approach that helped make the evaluation strategic was the emphasis on going beyond the simple tangible data. Many of the evaluation questions probed what lay underneath the tangible process and outcome data. For those in strategic roles, this was extremely important for building an understanding of why and how things were happening. With information about the underlying causes of effectiveness and ineffectiveness, the client is better able to design solutions and enhancements that will address the true, underlying causes rather than “window dressing” the indicators. In other words, strategic use is facilitated by conveying understanding, not just evaluative findings.
Another example of strategic evaluation comes from the work that the second author conducted with a regional healthcare system. One of the largest not-for-profit hospital systems in the US, leaders of this organization were acutely aware that strategic and successful succession planning was crucial to its future sustainability. Building a cadre of leaders capable of carrying the reins was important, but ensuring those individuals imbued a strong sense of the organization’s mission and values was imperative. Without holding the mission and values as a driving force, the organization risked becoming yet another healthcare system providing services to those in need in a cost-efficient way. Rather, the mission of this organization is to serve the poor. Therefore, decisions need to be made, and the hospitals and other types of healthcare facilities in this system need to operate daily, in a way that enables effective, appropriate service to the poor while functioning in a way that enables the organization’s fiscal health to remain strong. To meet their development needs, the organization sought a state-of-the-art process that would align the organization’s strategic priorities with five critical leadership factors (Jen – add these?) considered vital for its leaders.
The central intervention designed and delivered to address these strategic goals was a 15-month development process that included:
· A 3-day face-to-face orientation program
· Two 5-day feedback-rich, experiential leadership development programs
· Two phases of team-based action learning leadership project and coaching work[1]
· One-on-one coaching sessions
This intervention, named the Leadership Academy, was targeted at leaders who held hospital president, vice president, or similar positions. The roles they were being groomed to fill were at the regional CEO and executive team levels.
The needs assessment provided guidance for the design of the initiative as well as the design of the evaluation. Key stakeholders took part in a two-day intensive focus group session that provides a structure for first identifying the organization’s overall challenges and needs (e.g., to remain competitive in the market place, to develop a culture of ownership for the organization by employees) and linking leadership needs, leadership competencies, a conceptual design, and evaluation needs to them. Thus, stakeholders (the executive team in this case) identified <insert organizational needs> as the key organizational needs for which this leadership development initiative was required. The strategic evaluation questions were:
1. How effective are the action learning projects? In this case, do they a) provide a learning platform for participants and b) lead to actionable recommendations that are later adopted and put into action by the organization (and what are the outcomes of doing so?)?
2. How have participants enhanced the scope of their responsibilities (e.g., promotions, roles on organization-wide task forces)?