Submitted to:Submitted by LPO Student Team:

Debora Shiflett, MBAJeremy Covert

Director, VUMC Financial Training,Matt Dolson

VUMC Department of FinanceSamara Orkin

Ryan Seiberling

Sarah Sisk

Vanderbilt’s Finance Leadership Program Evaluation
November 27, 2012

Table of Contents

Executive Summary

Evaluation Introduction

Organizational Context of the Evaluation

The Evaluation Process

Purpose of the Evaluation

Key Questions Guiding the Evaluation

Evaluation Method

Stakeholders

FLP Logic Model

Evaluation Design

Data Collection Plan and Instruments

Data Collection Plan

Instrument 1 - Productivity & Satisfaction Survey (Pre/Post)

Instrument 2 - Reaction Survey

Instrument 3 - Outcome Survey

Instrument 4 - Focus Group

Coding Plan and Data Analysis

Instrument 1 - Productivity & Satisfaction Survey

Instrument 2 - Reaction Survey

Instrument 3 - Outcome Survey

Instrument 4 - Focus Group

Presentation Plan

Tables

Table 1: FLP CIRO Evaluation Methodology

Table 2: Evaluation Stakeholders

Table 3: Data Collection Plan

Table 4: Example Productivity & Satisfaction Survey Data Capture Table

Table 5: Example Results Survey Data Capture Table

Table 6: Example Outcome Survey Data Table (Likert Reponses)

Table 7: Example Outcome Survey Data Table (Question 8)

Table 8: Example Outcome Survey Data Table (Question 9)

Table 9: Example Outcome Survey Data Table (Question 10)

Table 10: Evaluation Results Communication Plan

Figures

Figure 1: Overview of the FLP’s Inaugural Offering

Figure 2: FLP Logic Model

Figure 3: Summary of Evaluative Design’s Annual Treatment Cycle

Figure 4: Example Productivity & Satisfaction Results Chart

Figure 5: Example Results Survey Data Results Chart

Figure 6: Example Outcome Survey Results Chart (Likert Reponses)

Figure 7: Example Outcome Survey Results Chart (Question 8)

Figure 8: Example Outcome Survey Results Chart (Question 9)

Figure 9: Example Outcome Survey Results Chart (Question 10)

Figure 10: Focus Group Coding Example

Executive Summary

Vanderbilt’s executive leadership team has avision to create a “One University” culture between the Vanderbilt Medical Center and the University that will ultimately require greater efficiency and communication across the organization. Therefore, the executive team determined that in order to drive the organization toward this new vision, finance leaders must possess and utilize leadership skills on the job and be able to skillfully foster a positive culture change – a primary component of which is “big picture thinking.”Succession planning was also prioritized for the finance department. As a result, Vanderbilt contracted The Advisory Boardto create andimplement a leadership development program, the Financial Leadership Program (FLP).

The FLP Evaluation Plan provides a process to assess if the program’sobserved impact aligns with the one intended. This plan will help to determine where and how program effectiveness may be improved, if at all. Using the following key questions, the evaluation plan will evaluate for:

  1. Leadership Skills: Has participants’ use of the tools and principles taught in the FLP improved work productivity and processes of the department?
  1. Big Picture Thinking: As a result of the FLP, do the participants find themselves better able to consider the potential impact of their decisions on the internal Vanderbilt clients they serve?
  1. Pipeline/Succession: Is the FLP program producing the leaders needed to fill the talent pipeline and meet the Finance Department’s future leadership needs?

The CIRO Evaluation Model was used to design an evaluative plan to assess the FLP’s ability to meet the needs established from the executive team which are reflected in the questions above.The evaluation follows the same year-long cycle of the FLP – commencing upon participant registration and ending four months after the FLP’s final application intensive.

Four evaluative instruments have been created and attached including two participant surveys which measure for reactions and program outcomes respectively. An employee productivity and satisfaction survey is providedto be given to direct reports of those attending the FLP, and a focus group interview was created for Cohort 1 participants. A data collection, coding, and analysis plan isincluded for each of the instruments to provide guidance in interpreting the instruments’ results in a manner that effectively addresses the key questions. The client may utilize the communication plan provided to report the evaluation results to the primary stakeholders.

Evaluation Introduction

Organizational Context of the Evaluation

Five years ago, a new Vanderbilt University executive leadership team came on board and began making changes. Recently, this executive leadership team, led by the ChancellorNicholas S. Zeppos, instituted a new initiative that would streamline processes and communications across the institution for greater efficiency and communication with the ultimate goal of operating as “One University.” The executive team determined that in order to drive the organization toward this new vision, finance leaders must possess and utilize leadership skills on the job and be able to skillfully foster a positive culture change – a primary component of which is “big picture thinking.” The executive team also expressed concern that individuals possess the required leadership skills needed to take over for the current leadership team when the time comes.To tackle these objectives, the senior executive team tapped our client Debora Shiflett, the Director of Vanderbilt University Medical Center (VUMC) Financial Training, to initiate a leadership development program for finance. (D. Shiflett, address to LPO class, September 25, 2012).(D. Shiflett, personal communication, October 14, 2012).

In addition to these new learning and cultural change objectives set forth by the senior executive team, VUMC’s finance staff has challenges inherent in their daily work. Specifically, the changing nature and complexity of health care regulations are external factors that consistently put pressure on leaders and workers to not just keep up, but to innovate. Furthermore, VUMC Finance Department is challenged to operate from a system’s perspective by creating and implementing financial systems across the institution. Having a “bigger picture” perspective is therefore important in both the operational and strategic work of the department (D. Shiflett, address to LPO class, September 25, 2012).

To meet the challenges issued by the senior executive team to create a leadership program for finance and in keeping with the reality of the current work environment,our client selected current vendor, The Advisory Board, to provide the content and facilitation for a new leadership development program. The Advisory Board was selected for multiple reasons: its research focus in healthcare, its reputation for implementing best practices for healthcare training, and its competitive cost structure because Vanderbilt is an existing customer. While the executive team continues to deliberate about leadership competencies needed for the finance staff, Debora benchmarked leadership skill requirements from well-known leadership competency models and decided that the competencies taught by The Advisory Board were a match with those needed by Vanderbilt’s finance team. The leadership program purchased is an established, pre-packaged product (D. Shiflett, personal communication, October 14, 2012). The curriculum includes a series of six workshops and three application intensives designed to focus on current culture challenges within the organization as well as on real-world work projects. The program was officially dubbed Vanderbilt’s “Finance Leadership Program” or FLP.

Debora and her leadership team recognized a unique opportunity to progress toward the desired culture of “One University” by including the University Central leaders in the FLP. As such, program participantsare from both the University Central and Medical Center Finance departments, the majority of which have never worked together prior to the start of the FLP. To further address cultural issues, The Advisory Board facilitator works closely with Debora to understand Vanderbilt’s unique culture and apply this knowledge to make workshop discussions and examples more relevant. In addition, certain components of the FLPhave been completely customized for Vanderbilt.

Objectives of the Finance Leadership Program

The next important milestone in the evaluation process is to clearly understand the objectives of the FLP. These were determined by the client in conjunction with The Advisory Board personnel. According to the FLP intranet website, the primary objectives of the program are to:

  1. Increase breadth and depth of leadership skills and organizational knowledge among current leaders
  1. Build new skills that will enable staff members to see, understand, and respond to the bigger picture and customers’ needs
  1. Develop staff for the next level of leadership - equipping them with the skills needed to take the place of existing leaders when the time comes

The FLP has the following components as illustrated in Figure 1 (D. Shiflett, address to LPO class, September 25, 2012):

  1. Six workshops administered over 7 months, including several prescriptive tools for implementing the concepts taught
  1. Three application intensive sessions designed to provide participants opportunities to practice the tools and concepts learned in workshops by applying them directly to real-world work projects
  1. 360 Feedback Instrument. Feedback is gathered on each participant from various perspectives (another Vanderbilt graduate group will be evaluating the 360/Coaching piece of the FLP)

Figure 1: Overview of the FLP’s Inaugural Offering

The learning methodologies deployed by The Advisory Board for the FLP include a combination of classroom learning, small group discussions, exercises and worksheets, case studies, application intensives where tools are practiced, and reading assignments (Shiftlett and Purinton, n.d).

The FLP objectives are in direct alignment with those identified by the executive leadership team to bring about the desired changes needed to move the organization closer to its change goals. After reviewing the program components they appear to be well-suited to help participants be able to transfer leadership learning and skills back on the job, to promote cultural change, and to help fill the leadership pipeline. An evaluability needs assessment suggests that the FLP is well positioned to be evaluated according to the stated needs of the client.

The Evaluation Process

Purpose of the Evaluation

Our purpose with this Vanderbilt Finance Leadership Program Evaluation Plan is to provide a process the client may use to assess the FLP to determine if the program’s actual impact aligns with the one intended. This evaluation plan will help to determine where and how program effectiveness may be improved, if at all. Specifically, the evaluation plan will look at several aspects of the program’s impact including but not limited to:

  • The value of the concepts and tools presented to the participants
  • If the participants actually implemented the tools into their own departmental culture
  • What changes occurred as a result of implementation
  • How participants were able to apply concepts from the program
  • How did the use of the tools and concepts improve productivity, processes, and work culture

Closely reviewing the impactof the workshops and application intensives, getting feedback from participants as well as other stakeholders, and repeating this evaluation as a part of each annual offering of the FLP willassist Finance Department leadership with understanding the FLP’s impact.

Key Questions Guiding the Evaluation

The client initially provided several questions she wanted answered about the FLP’s effectiveness.After thoughtful consideration, our group produced three primary focus areas around which these questions seemed to cluster: leadership skills, big picture thinking, and pipeline/succession planning. We then distilled the questions within these categories into a key question which will be used in guiding our evaluation plan.

  1. Leadership Skills: Has participants’ use of the tools and principles taught in the FLP improved work productivity and processes of the department?
  1. Big Picture Thinking:As a result of the FLP, do the participants find themselves better able to consider the potential impact of their decisions on the internal Vanderbilt clients they serve?
  1. Pipeline/Succession:Is the FLP programproducing the leaders needed to fill the talent pipeline and meet the Finance Department’s future leadership needs?

Evaluation Method

Vanderbilt’s FLD Program will be evaluated using the CIRO Approach posed by Warr, Bird, and Rackhamwhich frames the evaluation around context, input, reaction, and outcomes (1970). This approach is particularly applicable for leadership development programs and for new programs, both of which are the case here. The CIRO Approach fits well because the client has expressed larger organizational objectives that the program should meet, and CIRO frames an evaluation first with the larger organizational objectives desired. In other words, the first step is to unearth what is lacking, what would fill the gap, and what goals the organization has created that created a need for the intervention in the first place. From this contextual frame, all other inquiries flow down from there, including questions about inputs, participants’ reactions, and outcomes. The CIRO Approach delivers additional value because it allows outcomes to be presented first, thus allowing usto quickly report results that impact the client’s overarching organizational goals.

In order to address the FLP’s essential evaluation questions, our use of the CIRO Approach will involve gathering data around the following:

Table 1: FLP CIRO Evaluation Methodology

FLP CIRO Evaluation Methodology
Context /
  • What is missing, what is needed, and are the leadership development goals of the Finance Department(FD) being addressed by The Advisory Board FLP?
  • Is the FLP comprehensive enough to meet all of FD’s leadership goals?

Input /
  • What leadership resources (i.e. personnel) are available as a result of the program effort?
  • What supports are necessary to reinforce the changes after the program is completed?

Reaction /
  • What do participants perceive regarding the FLP’s relevance and timeliness?
  • What do participant perceive regarding their ability to apply skills learned back on the job?

Outcome /
  • Did the FLP sufficiently increase the leadership skills, the big picture thinking capabilities, and the leadership succession pipeline of the joint finance department managers?

Stakeholders

The following tables provide a breakdown of the primary and secondary stakeholders of the FLP. Based upon conversations with our client, we recognize the primary stakeholders as those persons with significant influence and decision-making authority. They have direct impact on the program and will be the major users of the evaluation data. The secondary stakeholders are those directly and indirectly involved with the program with no decision-making authority, as well as, those persons who will indirectly feel impacts of the FLP.

Table 2: Evaluation Stakeholders

Primary Stakeholders (Program Decision Makers)
Stakeholder / Role
Brett Sweet / VU CFO
Eric Kopstain / Associate Vice Chancellor for Finance
Warren Beck / Associate Vice Chancellor for Health Affairs for VUMC
Cohort 1 Participants / Senior Managers and Managers receiving training
Debora Shiflett / Director, VUMC Financial Training & Primary Evaluation Client
Secondary Stakeholders
Stakeholder / Role
Cohort 2 Participants / Additional managers receiving training
Non-participating Finance Department employees / Frontline customer interactions
Finance Department Customers / Recipients of improved performance

1 | Page

FLP Logic Model

A key tool known as a logic model was designed to gain a clearer understanding froma systems perspective of the FLP program. A logic model is especially useful in making explicit connections between important program components, specifically, the FLP’sresources, activities, outputs, and impacts. By reading left to right, the reader may add the words, “if this, then” to understand how each component of the program directly feeds the next category.

Figure 2: FLP Logic Model

1 | Page

Evaluation Design

In crafting a design for the FLP Evaluation, our goal was to link the design to the specific needs of the client and to ensure the design of the evaluation fits with the methodology chosen. The evaluation design provided in Figure 3is consistent with the CIRO approach in that it will attempt to address the Primary Stakeholder’s key questions within a framework of what the FLP may be lacking (Context), the leadership personnel resources available to the Finance Department as a result of the program (Input), the participants’ perception of the sufficiency/applicability of skills gained (Reaction), and the actual impact the program appears to be having (Outcomes). See section Data Collection Plan and Instruments for specific questions associated with each level of the CIRO model, collection instruments used to assess for them, and the timing for administration.

The following are key components and factors impacting the evaluation’s design:

Structure / Timing

  1. The evaluation design views the FLP as a single annual treatment consisting of nine “component treatments” (six workshops / three application intensives) conducted within the first seven months of the annual treatment cycle.
  2. The evaluation design uses a mix methods approach consisting of pre/post-test survey design (productivity & satisfaction survey), time series design (reaction survey), and a post-treatment assessments (outcome survey and focus groups). This design will produce both quantitative and qualitative data, the analysis of which is discussed in detail in the Coding Plan and Data Analysissection.

Sampling

  1. 100% of employees reporting directly to managerial program participants will be given the pre/post treatment Productivity & Satisfaction Survey
  2. 100% of program participants will be given the Reaction Surveys and Outcome Survey
  3. Sampling for the Focus groups will consist of all Cohort 1 participants. Groups should not consist of more than twelve participants each.

Use of Results