Massachusetts Race to the TopCollege and Career Readiness Initiatives
Evaluation Annual Report, September 2012
Prepared for the Massachusetts Department of Elementary and Secondary Education
/ UMass Donahue InstituteResearch and Evaluation Group / 1
RTTT C&CR Evaluation Annual Report, September 2012
Acknowledgements
The UMass Donahue Institute extends its sincere appreciation to the many people who supported and collaborated with us on this evaluation. In particular, we want to thank personnel from the Massachusetts Department of Elementary and Secondary Education and from the schools, districts, partners, and vendors associated with the Massachusetts Pre-AP, STEM Early College High School, and MassCore initiatives.
Massachusetts Race to the TopCollege and Career Readiness Initiatives
Evaluation Annual Report, September 2012
Project Staff
Eliot Levine, Senior Research Manager, Project Manager
Jeremiah Johnson, Research Manager
Patricia Lee, Research Analyst
Steven Ellis, Director, Research and Evaluation
Report Information
This report was prepared by the UMass Donahue Institute, the project evaluator, under contract with the Massachusetts Department of Elementary and Secondary Education.
About the Donahue Institute
The University of Massachusetts Donahue Institute is the public service, outreach, and economic development unit of the University of Massachusetts President’s Office. Established in 1971, the Institute strives to connect the Commonwealth with the resources of the University through services that combine theory and innovation with public and private sector applications.
UMDI’s Applied Research and Program Evaluation group specializes in applied social science research, including program evaluation, survey research, policy research, and needs assessment. The group has designed and implemented research and evaluation projects for diverse programs and clients in the areas of education, human services, economic development, and organizational development.
University of Massachusetts Donahue Institute413-587-2400 (phone)
Applied Research and Program Evaluation Group413-587-2410 (fax)
100 Venture Way Suite 5
Hadley, MA 01035-9462
Table of Contents
Acknowledgements
Executive Summary
Introduction
Pre-AP Training and Alignment
STEM-focused Early College High Schools
MassCore Policy and Implementation
Appendices
Appendix A1: Pre-AP Field Interview Protocol, Teacher or Administrator...... 97
Appendix A2: Pre-AP Teacher and Lead Teacher Survey...... 101
Appendix A3: Pre-AP Administrator Survey...... 112
Appendix A4: Pre-AP Vendor Interview Protocol...... 119
Appendix A5: Pre-AP Vertical Team Meeting Observation Protocol...... 122
Appendix A6: Pre-AP Teacher Summer Training Observation Protocol...... 123
AppendixB1: ESE Interview Protocol...... 124
Appendix B2: STEM ECHS JFF Interview Protocol...... 126
AppendixB3: STEM ECHS Pre-Implementing Schools Interview Protocol...... 128
AppendixB4: STEM ECHS Implementing Schools Interview Protocol...... 131
AppendixB5: STEM ECHS IHE Partner Interview Protocol...... 134
AppendixB6: STEM ECHS School Personnel Focus Group Protocol...... 136
AppendixB7: STEM ECHS Observation Protocol...... 138
AppendixC1: MassCore Administrator Interview Protocol...... 139
AppendixC2: MassCore District Administrator Survey...... 142
Executive Summary
The Massachusetts Department of Elementary and Secondary Education (ESE) was awarded a federal Race to the Top (RTTT) grant for the years 2010-2014. The college and career readiness (C&CR) components of the grant were designed to help students develop "knowledge and skills necessary for success in postsecondary education and economically viable career pathways." The RTTT initiatives are intended to provide students with opportunities to participate in quality, upper-level high school coursework and new approaches to assist them with high school completion and transition to higher education and the workforce. These programs are part of a broader effort, as specified in the Delivery Plan of ESE's College and Career Readiness group, to increase the Massachusetts5-year high school graduation rate to 88.3% and the number of students who complete the MassCore program of study to 85% statewide.
The UMass Donahue Institute is conducting an evaluation of three of the C&CR components of the Massachusetts RTTT efforts – the Pre-AP Training and Alignment Program, the STEM-focused Early College High Schools (ECHS), and the MassCore Policy and Implementation initiative.
For each of these three programs, this executive summary provides a brief program description, followed by evaluation findings for the period ending September 30, 2012 (referred to as Year 2). Strategic considerations are also presented for some programs. The full report that follows the executive summary incorporates findings from all evaluation activities conducted to date, including a synthesis of the three previous quarterly evaluation briefings. A companion document, the Technical Supplement – RTTT C&CR 2012 Annual Evaluation Report, presents tables and figures that provide a more complete summary of baseline graduation and dropout data.
Pre-AP Training and Alignment
The aims of the Pre-AP Training and Alignment program are to increase the number of low-income and minority students prepared to participate and succeed in mathematics, science, and English Advanced Placement courses and credit-bearing college-level coursework; to provide teachers in grades 6-12 with high-quality professional development to assist them in developing curricula, instruction, and assessments that prepare students for AP coursework; and to provide an opportunity for teachers to collaborate in horizontal and vertical teams and to network with other teachers in their region for the purpose of improving curriculum and instruction. Technical assistance for the project is based on the Laying the Foundation (LTF) curriculum and is provided by the Massachusetts Math + Science Initiative (MMSI), a project of Mass Insight Education in partnership with the Commonwealth of Massachusetts.
Districts are participating in Pre-AP training in multiple configurations. The configuration for RTTT Goal 4D is to send the same teachers for four-day trainings in three consecutive summers and to create discipline-specific vertical teams, led by trained lead teachers, that meet quarterly to share and deepen their Pre-AP practices. Districts that are using Goal 4A or non-RTTT funds have created several variations of the basic training configuration, and the percentage of all teachers in a given school and district who are being trained varies widely. Districts also selected different disciplines or combinations of disciplines among the mathematics, science, and ELA trainings offered.
The Year 2 evaluation of the Pre-AP initiative included the following activities and data sources: interviews of teachers and administrators; surveys of teachers, lead teachers, and administrators; interviews of ESE Program Manager and Project Director; interviews of technical assistance vendor; observations of vertical team meetings; observations of Pre-AP training sessions; training registration and attendance database; vertical team meeting attendance database; supplemental school data requests; ESE documents and databases; and, related project documents.
The Pre-AP Training and Alignment initiative is well underway. During summer 2011, MMSI provided four-day summer trainings for 561 teachers from 37 districts (including 19 4D districts), and teachers attended an average of 3.6 days (out of 4). Also during summer 2011, 30 teachers participated in lead teacher training, and 25 administrators (from seven 4D districts) participated in administrator training. For summer 2012, more than 900 teachers registered for training; final registration data were not available for this report. The ESE program manager reported that, after summer 2012, Massachusetts will have met the RTTT performance goal of having trained 1,000 teachers. In addition, the technical assistance and support provided by MMSI and ESE were widely seen as helpful and satisfactory.
Lead teachers and administrators generally reported being satisfied with the Pre-AP training events. Administrators and teachers largely agree that the LTF lessons and assessments are well-aligned with the Common Core standards and are examples of high-quality pedagogical practices. Teachers were asked about the impacts of their participation in Pre-AP training, and 70% or more of 4D teachers agreed that, as a result of the Pre-AP training, they now teach more Pre-AP content, use more Pre-AP pedagogical strategies in the classroom, and have greater awareness of the importance of using Pre-AP strategies.
Substantial challenges to Pre-AP implementation were reported by both administrators and teachers with regard to curricular resources, classroom equipment, planning time, pacing guides, and student academic readiness. Such concerns represented a substantial minority in some cases, and a clear majority in others, particularly in relation to the adequacy of planning time and the academic readiness of students for the Pre-AP curriculum. The number of Pre-AP lessons and assessments that teachers are expected to implement varies widely across districts.
During the 2011-12 school year, 4D district teams reported attendance at an average of 3.5 vertical team meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer 2011 trainees, plus an average of four personnel who hadn’t completed Pre-AP training. Some lead teachers were not satisfied with the agendas and activities provided by MMSI to structure vertical team meetings. Some lead teachers made slight modifications to agendas, and others completely restructured agendas to meet their district’s needs. The MMSI program director reported encouraging lead teachers to make changes as needed to meet their school’s goals.
The LTF program philosophy, strongly shared by ESE, is that every student should be considered a Pre-AP student, but evaluation findings indicate diverging voices and practices on this topic in the field. Most teachers and administrators who were interviewed agreed that every student should be considered a Pre-AP student, and multiple interviewees expressed mild to strong opposition to tracking. In contrast, responses to the teacher survey provided more varied responses. The survey asked teachers if they used more Pre-AP materials and strategies with some classes than with others. About half said yes, and most of those indicated that they used more Pre-AP materials and strategies with their honors or advanced-level students. This group of respondents generally agreed that Pre-AP materials were not suitable for all students, such as those in remedial classes, those who had not mastered skills from a previous course, and some English language learners.
Strategic considerations (explained in greater detail in the full report).
- Expectations for minimum implementation levels should be clarified or reiterated.
- Providing additional materials and suggestions for differentiation of LTF lessons may increase classroom implementation.
- Further assessments of MMSI trainer quality, and additional training of some trainers, may be indicated.
- The level of implementation of Pre-AP practices should be monitored and interpreted in relation to student impacts.
- Pre-AP schools may need to budget for additional equipment and professional development.
- Clarification of the intended uses of vertical team meeting time might lead to increased Pre-AP implementation.
- Sharing of modified vertical team meeting materials could save lead teacher time and enhance meeting productivity.
STEM-focused Early College High Schools
In its Race to the Top proposal, ESE proposed to open six STEM early college high schools to prepare students for productive STEM careers, to reduce achievement gaps, and to provide an accelerated pathway to postsecondary education for underrepresented students by partnering with colleges and providing opportunities to earn up to two years of college credit while still in high school. Six districts were chosen in a competitive process and are currently receiving RTTT Goal 4E funds for this purpose. Eight additional STEM ECHS sites in the state are being established, most with the support of discretionary RTTT funds. These additional sites will be included in future evaluation activities, but ESE has requested that UMDI wait until the work of those sites has progressed further.
Evaluation activities during Year 2 included interviews with the ESE Program Manager for STEM ECHS, technical assistance vendor, administrators, and IHE partners; a focus group with school personnel from one implementing site; two site visits, and analysis of relevant documents and databases. Baseline graduation and dropout data are presented for STEM ECHS schools and districts.
The six Goal 4E STEM ECHS sites are in various phases of project development and implementation. Two sites enrolled students in STEM ECHS activities during the 2011-12 school year, three sites plan to start during the 2012-13 school year, and the last site’s STEM ECHS work has been incorporated into a broader school redesign plan.
Interviewees from five districts described assembling a planning team as a successful aspect of their planning process. Additional factors that contributed to the success of the planning teams were including representatives from diverse stakeholder groups and consistent collaboration across members. Respondents shared that holding planning meetings on a regular basis was difficult, because most team members had many other substantial responsibilities and commitments. Interviewees from all districts cited support from district leadership as essential to the planning process and establishing strong working relationships with their IHE partner.
Informants reported many challenges that arose during the planning phase, including budget management, instruction and curriculum development, staffing, engaging the community, and working with teachers’ unions. Several districts also reported challenges related to the quality, frequency, or clarity of communication across stakeholders.
Marlborough is the only site that began full implementation during Year 2, with 189 students in grades 6 and 9. One Marlborough interviewee said that a primary success has been the staff’s ability to encourage and support all students in their efforts to complete a curriculum of five honors-level courses. All teachers said that the majority of STEM ECHS students were having a positive experience and “moving in the right direction.” The interviewees expressed that the following factors have contributed to their successful implementation to date: strong curriculum development, a collaborative partnership with the IHE, and the district’s commitment to providing teachers with time for professional collaboration.
Many challenges were reported by Marlborough teachers. One high school teacher reported that some colleagues working within the main high school resent the increased resources (e.g., structured planning and collaboration time, professional development) and decreased class sizes enjoyed by the STEM ECHS teachers. A middle school teacher reported that building administrators assigned personnel to the STEM ECHS without giving them the opportunity to opt out of the program. STEM ECHS teachers also reported that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative, which was not fully implemented until five months into the school year. Teachers said that it was difficult and time-consuming to integrate the use of laptops into the curriculum at that point in the year.
All districts reported that ESE had consistently provided effective, professional, and timely assistance. Districts noted that ESE personnel were aware of, and sensitive to, the contexts in which the districts were working. Jobs For the Future, working in consultation with ESE, provided technical assistance to the Goal 4E STEM ECHS sites. The support provided to the sites has individual technical assistance, meetings with all six schools, and “just-in-time” problem solving. The majority of districts said that JFF was viewed as a valued partner that was helping to move their programs forward.
All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) reported that finding the financial resources to sustain these STEM ECHS sites will be a significant challenge. No districts have established a clear plan for addressing this issue, although several said that they anticipated committing increased levels of attention to sustainability issues over time.Oneinterviewee also suggested that, over time, the state could consider adopting policies that would benefit or expedite the development of STEM ECHSs and other programs providing high school students with early college experiences.
Strategic considerations (Explained in greater detail in the full report).
- Initial lessons from the early-implementing STEM ECHS schools could benefit the other STEM ECHS schools and inform technical assistance from JFF and ESE.
- Continuity of leadership and succession planning appear important to success of the STEM ECHS schools.
- Securing the financial resources to sustain STEM ECHS partnerships is a primary concern of all STEM ECHS schools.
- Changes to state and local policies may facilitate STEM ECHS development.
- Increased connections with the broader STEM community could benefit the STEM ECHS schools.
MassCore Policy and Implementation
The Massachusetts High School Program of Studies (MassCore) recommends a set of courses and other learning opportunities which Massachusetts students should complete before graduating from high school, in order to arrive at college or the workplace well-prepared and without need for remedial coursework. The 155 districts that selected the RTTT college and career readiness goal committed to implementing strategies to increase the percentage of their students who complete the MassCore curriculum. The state’s RTTT goal is to increase the statewide MassCore completion rate from the 2010 baseline of 70% to 85% of graduates by 2014, and the state has created a formula-based goal for each district. Each district was also expected to determine areas in which courses or supports needed to be expanded in order to meet the 2014 targets, and to create and implement a plan to improve the accuracy of their reporting of MassCore completion levels.
Evaluation activities during Year 2 included an interview with several RTTT district administrators, a survey of RTTT district administrators, interviews with the ESE Program Manager for MassCore and ESE Project Director for the RTTT C&CR evaluation, and analysis of ESE documents and databases. Baseline graduation and dropout data are presented for RTTT MassCore districts.