Principal Leadership Institute

Principal Leadership Institute:

Preparing the next generation of urban school leaders

Evaluation Report #1

Janet Lee and Tanner LeBaron Wallace

SRM Evaluation Group

Graduate School of Education and Information Studies

UCLA

October 2004

In the spring of Spring of 2004 members of the Social Research Methodology (SRM) Evaluation Group designed an evaluation proposal for the Principal Leadership Institute (PLI) at the University of California Los Angeles (UCLA). The UCLA PLI Leadership Team felt a need to develop an internal system for on-going data collection and evaluation in order to gather information regarding student demographics, PLI program processes, student knowledge, and participant perceptions. Based upon the SRM Evaluation Group’s proposal, this internal data collection system creates a mechanism for PLI stakeholders to monitor progress towards important outcomes over the fifteen months of student training for each cohort. The evaluation design permits for cross-cohort analysis, intra-cohort analysis, as well as the understanding of individual development as the program progresses. With the commencement of a fifth cohort of students in June of 2004 and the graduation of a fourth cohort of students in August of 2004, the SRM Evaluation Group began collecting empirical information to measure PLI’s success in accomplishing its mission to develop a cadre of transformational urban school leaders prepared to create learning communities where all students achieve an excellent education and have the choice to attend college. Key evaluation questions focus on the entire PLI process from recruitment to preparation to on-the-job effectiveness.

Introduction to Evaluation

In this section we provide a review and overview of our proposed evaluation before going into detail about activities, methodology and results. Our approach to evaluation is a utilization-focused process. We believe the evaluators’ role is to facilitate a process that will yield maximum utility to the intended users of the evaluation. Therefore, we will collaborate with the PLI Leadership Team throughout the course of this evaluation to ensure that questions asked, data collection strategies used, and information provided will meet program needs.

From extended conversations with the Leadership Team, it is our understanding that PLI is ready to take a broad look at the program in order to eventually measure overall effectiveness and to plan for future development. The overarching question being asked is: To what extent is PLI successful in its mission to develop effective change agents in urban schools? Based upon these information needs, our evaluation focuses on the following four areas:

• Creating a system for on-going, internal data collection;
• Gauging the experiences and development of PLI students over time;
• Locating areas for improvement within the PLI program; and
• Ascertaining the effectiveness of the program.

We proposed conducting a staged evaluation in order to meet the program’s immediate and long-term needs.

Stage 1: Building Evaluation Infrastructure – Currently in progress

1. Assess the current state of the program– Develop the Theory of Action

The first step in the evaluation was to assess the current state of the program. We began by reviewing all available documents that either described the PLI program and/or stated the goals/purposes of PLI. However, after this review, we found that the information provided in the documents proved to be an insufficient starting place for developing an understanding of the mechanisms behind the workings of the program. Therefore, in order to gain this working understanding of the PLI program, we engaged the intended users of the evaluation (the Leadership Team) in a series of discussions to articulate PLI’s ‘Theory of Action,’ the logic behind the structure of the program. Based upon this important initial step of the evaluation, the evaluators along with the primary-intended users are able to hypothesize relationships between important aspects of a program theory such as inputs, activities, immediate outputs, intermediate outcomes and ultimate goals. In essence, PLI’s Theory of Action, developed and created in collaboration with the Leadership Team, links important program components in relationships that demonstrate ideally how the program intends to achieve its ultimate goal of developing effective change agents in urban schools. PLI inputs, activities, immediate outputs, intermediate outcomes and ultimate goals are mapped out and linked to form a PLI program theory (see Appendix A). This theoretical model serves to focus the efforts of key stakeholders. All of the outlined relationships must be present in order to accomplish the ultimate goal which, for PLI, is a principal training program grounded in social justice theory working to produce leaders able to transform schools into places where every student has the opportunity to learn and succeed. Moreover, the Theory of Action provides the evaluation team the all-important “compared to what” of the evaluation. As the idealized version of the PLI program, the model informs the evaluators what data to collect as well as how to subsequently analyze the data.

The components of the model were determined in conjunction with the PLI Leadership Team. Several key discussions with the PLI Leadership Team were then held to clarify the linkages between each of the model components. The importance of establishing PLI’s Theory of Action cannot be understated. Identifying the logic behind how the program intends to achieve its ultimate goal not only focuses the all program stakeholders in order to accomplish the program mission, but also invaluably assists the evaluation process by providing a point of comparison.

1. Establishing a web-based system for evaluation

A primary goal of the PLI evaluation is to develop a web-based, internal data collection system to gather on-going information over time. This system should be low-maintenance and easily accessible to a variety of stakeholders. The data collected over time will help PLI determine: Whom does PLI recruit? How is PLI currently operating? And, where does PLI need to change in order to accomplish “the mission” as outlined by the Theory of Action? Several survey software options have been presented to the Leadership Team. However, the introduction of the Moodle course management system to PLI in November 2004 may eliminate the need for a separate software application. One of the Moodle modules can be adapted in order to administer secure online surveys to cohort members. The module will also organize data, perform basic statistical analysis, and permit raw data to be exported to other applications. Working in conjunction with Jovcic Slobodan of the Educational Technology Unit, the evaluation team plans to administer the first on-line survey to Cohort V in February 2005. From that point on, all surveys in the series will be administered through the secure PLI course management system.

Stage 2: Formative Evaluation – Started June 2004

1. Internal data collection activities

The evaluation team began collecting data in June of 2004. We developed a series of five surveys to be administered to PLI students at predetermined points during the PLI program and beyond. The dates have been purposefully chosen in order to provide longitudinal data. This system will allow for cross-cohort comparisons, intra-cohort comparisons, as well as the ability to follow individual students as they develop during the program. The proposed dates of administration for these surveys, as well as what data will be collected can be seen in Table 1, on the next page.

Table 1: Surveys to be administered to each cohort of PLI students

Survey No. / Date of Administration / Data Colleted
1 / April before enrollment /

• Recruitment/enrollment information
• Application process feedback
• Professional experience
• Career aspirations
• Attitudes and beliefs regarding urban schools
• Self-assessment for tolerance of risk and challenge
• Demographic information

2 / August after summer session /

• Initial impressions of PLI program
• Feedback on summer session program components
• Attitudes and beliefs regarding urban schools
• Demographic information

3 / February mid-point of program /

• Feedback on PLI program
• Curriculum content knowledge
• Career aspirations
• Attitudes and beliefs regarding urban schools
• Self-assessment of tolerance for risk and challenge
• Demographic information

4 / August graduation /

• Career aspirations
• Attitudes and beliefs regarding urban schools
• Self-assessment of tolerance for risk and challenge
• Self-assessment regarding leadership preparedness
• Curriculum content knowledge/leadership

Scenario prompt

• Feedback on PLI program components
• Demographic information

5 / One year after graduation August /

• Satisfaction with PLI leadership preparation
• Suggestions/reflections on PLI program components
• Attitudes about urban school transformation
• Current position
• Career aspirations
• Demographic information

Thus far, Cohort IV was administered Survey 4 via paper and Cohort V was administered Survey 1 via paper. Ideally, Survey 1 would have been administered to Cohort V prior to enrollment. Since the evaluation had just begun at that point, it was not possible. Therefore, it was decided to administer Survey 1 in August with some additional Survey 2 questions included. Cohort V will receive Survey 3, 4, and 5 on schedule. Cohort VI will be the first Cohort to receive all 5 surveys at the appropriate time intervals. Data analysis for Survey 1 and 4 has been completed. Briefly, Survey 1 is intended to gather recruitment and enrollment information as well as baseline data regarding career aspirations and attitudes towards urban school transformation, Survey 2 is intended to capture initial impressions about the PLI program and to gather feedback on the summer session, Survey 3 gathers feedback information on the PLI program as a whole and data regarding career aspirations and attitudes towards urban school transformation, Survey 4 assesses curriculum content knowledge and leadership preparedness, and Survey 5 asks PLI alumni to reflect on PLI program components as well as satisfaction with the leadership preparation. The full reports for each follow in subsequent sections.

Survey1 (Cohort V) Analysis

The evaluators designed Survey 1 to capture baseline data from incoming PLI students. Key questions of the PLI Leadership Team were included as major data topics in the survey. Information was collected from cohort members regarding the application process, details of professional experience, career aspirations, attitudes and beliefs regarding urban schools, tolerance of risk and challenge, and demographic information. This survey was administered by paper to Cohort V during the last session of their summer course in August 2004. If students were absent, a follow-up survey was mailed with a return date requested. We had almost 100% participation, with 41 of 42 surveys being returned. This section provides a detailed analysis of each question on the survey. We have organized the report by the major data categories of the survey. Tables with exact frequencies and percentages of responses can be found in Appendix B.

Results

Sample Demographics

Approximately two-thirds of Cohort V is female, and on average, 33 years old. Approximately one-third of the students are Euro American/white, one-third are Latino/Hispanic/Chicano, and the last third are Asian American, African American, or identified themselves as ‘Other.’ (see Table 1, Appendix B)

Recruitment and enrollment

Students were asked three questions regarding recruitment and enrollment. For these questions, students were able to choose more than one response. Therefore, each response has its own percentage totaling 100% and the percentages for each question will collectively add up to more than 100%.

When students were asked how they initially heard of PLI, an overwhelming majority of students’ initial contact with the program was through a personal connection, whether it be a friend, colleague or faculty member. When looking at the total number of responses, which is not the same as looking at the total number of students, we see that 80% of the responses indicate initial information about UCLA’s PLI was gathered from a personal connection. The other noteworthy means by which students learned of the program is through the Internet (23% of cohort members). (see Table 2, Appendix B)

Students were then asked what helped in their decision to apply to the program. Once aware of PLI, the PLI website made a strong contribution to the decision to complete an application to the program (45% of students). However, it was the on-site information session that had the strongest reported influence on students’ decision to apply (65% of students). (see Table 3, Appendix B)

The third question in this section asked for what reasons students decided to enroll in the PLI program. Once accepted into the program, students indicated many reasons equally contributed to their ultimate decision to enroll in the PLI training program. In fact, for five of the thirteen response choices more than 50% of the cohort indicated that particular factor contributed to their decision to enroll. Seventy-five percent of students indicated that PLI’s association with UCLA was an influence on enrollment. Additionally, other program components, such as length of program, the Tier 1 administrative credential, Masters of Education degree, curriculum, social justice orientation and cohort model also made contributions to students’ decision to enroll. The one non-program factor that notably influenced a substantial amount of students’ to enroll was the recommendations from PLI students or alumni, once again confirming the importance of personal contact in the recruitment process. (see Table 4, Appendix B)

Professional Experience

On average, members of Cohort V had been teaching for 7 years when they enrolled in the program, but classroom teaching experience ranged from 0 to 20 years. Just about half of PLI students in Cohort V are currently in classroom teaching positions. The other half of PLI students in Cohort V hold various coaching, coordinating, and counseling positions. Almost all currently have positions in urban settings and just over half have had some experience with non-credential graduate classes prior to entering PLI. (see Table 5, Appendix B)

Career Aspirations

As a principal training program affiliated with a well-known graduate school of education, questions regarding students’ career aspirations were of interest to the evaluation users. In order to determine career aspirations, students were asked to what extent they agreed/disagreed with the statement “My main career objective is to become an urban school principal.” Approximately 80% of students either ‘agreed’ or ‘somewhat agreed’ with this statement. Of the remaining students, 6 of 41 (15%) were ‘undecided,’ and 2 of 41 (5%) ‘Somewhat disagreed’ with the statement. There were no respondents who answered ‘strongly disagree.’ Of those who responded ‘agree’ or ‘somewhat agree,’ 57% indicated they aspire to be an elementary school principal, 12% indicated they aspired to be a junior high principal, and 20% indicated they aspire to be a high school principal. When subsequently asked where students saw themselves in 3-5 years, the overwhelming majority (85%) of all Cohort V members saw themselves either in an assistant principal or principal position. (see Tables 6-8, Appendix B)

Attitudes and Beliefs regarding Urban Schools: Questions 8 - 11

Students were asked how often in their current job they witness practices and policies that prevent every student from having an equal opportunity to choose attending college. The majority of students in Cohort V responded that they at least ‘sometimes’ observe practices and policies that prevent every child from having an equal opportunity to choose attending college. Of the PLI students in Cohort V, 97% felt that urban schools need to change ‘a lot’ or ‘completely.’ The students were asked to provide a short answer to the question “What changes need to be made in urban schools?” There were a total of 115 changes suggested in the student responses. The responses formed 20 qualitatively different categories. Seventy-five percent of the total responses from the ten highest frequency response categories indicated strong similarities in responses throughout the cohort. This could, in part, be due to the administration of the survey after summer session coursework. All cohort members were exposed to similar course content. Ideally, responses to these questions would be compared to data from surveys administered prior to PLI enrollment and to data from surveys administered after summer session. Frequencies of response categories are detailed in Figure 1 in Appendix B. Of the categories that emerged, the four that received the highest frequency of responses were (in order): school procedures, attitude changes among staff, teacher quality/retention, and expectations for student learning. When asked about how challenging it is to lead and initiate change, almost all agreed that it was at least ‘somewhat challenging.’ (see Tables 9-11 and Figure 1, Appendix B)

Tolerance of risk and challenge

When asked how comfortable they were with risk and uncertainly, students indicated that they were at least ‘somewhat comfortable’ with risk and uncertainty. However, when asked how capable they felt leading the process of urban school transformation, only 25% felt ‘capable’ or ‘very capable.’ This might be expected, however, since students are at the beginning of the training program. Over the fifteen months in the PLI training program, we would expect this percentage to increase dramatically and near 100% at the time of graduation from the program. (see Tables 12-13, Appendix B)

Survey4 (Cohort IV) Analysis

Survey 4 was designed by the evaluators to capture information from cohort members regarding curriculum content knowledge, career aspirations, attitudes and beliefs about urban schools, self-assessed perceptions of leadership preparedness, PLI program components, and demographic information. Because Cohort IV was not administered surveys 1, 2 and 3, the Leadership Team was interested in collecting recruitment/enrollment and professional experience information as well. Therefore, recruitment/enrollment and professional experience questions were included in this particular survey. (This information is not included under the Survey 4 template in Table 1.) Like Cohort V, members of Cohort IV were administered a paper survey on campus during the last session a summer course. Because of absences and time constraints, several students were unable to complete the survey. Therefore, as was the case with Cohort V, a second attempt to gather as much data as possible was made by mailing out a copy of the survey with instructions asking students to mail back the survey. We received complete surveys for 37 of the 46 students in Cohort IV. Expectation is a 100% response rate with the advent of the online data collection system. This section provides a detailed analysis of each question on the survey. Again, we have organized the report by the major data categories of the survey. Tables with exact frequencies and percentages of responses can be found in Appendix C.