Massachusetts School Redesign Grant Initiative
Final Evaluation Report
Key Findings
Submitted to the Massachusetts Department of Elementary and Secondary Education
June 2015
UMass Donahue InstituteResearch and Evaluation Group
Acknowledgements
The University of Massachusetts Donahue Institute wishes to acknowledge the contributions of the many people and organizations from across the Commonwealth of Massachusetts who participated in or otherwise supported this study of the Massachusetts School Redesign Grant initiative. Their collective efforts in support of school redesign, and this study, were truly inspirational and deeply appreciated.
In particular, the study team wishes to thank the district leaders, school leaders, and school-based educators who gave so generously of their time, sharing their insights and first-hand experience with the complex, challenging, and utterly essential work of school redesign. This report is a testament to their commitment both to enacting change and to contributing to greater understanding of their redesign process. We wish them all great success in their continuing efforts to build the most effective schools possible for the Commonwealth’s students.
The study team would also like to recognize leaders and staff of the Massachusetts Department of Elementary and Secondary Education (ESE), who supported, challenged, and learned from this study. The deep commitment of ESE’s Office of District and School Turnaround was evident from the inception of the initiative, and the knowledge of School Redesign Grant Coordinator Erica Champagne, in particular, was critical to the study. Similarly, staff of ESE’s Office of Planning andResearch, most notably Kendra Winner and Gerrie Stewart, served as indispensable colleagues, supporting the study’s conceptualization and implementation.
UMass Donahue InstituteResearch and Evaluation Group
Project Staff
Project Staff
Greta Shultz, Senior Research Manager
Sue Leibowitz, Senior Research Manager
Ivana Zuliani, Research Manager
Molly Fenton, Research Analyst
Steven Ellis, Director, Applied Research and Program Evaluation
This report is prepared by the project evaluator for the Massachusetts School Redesign Grant (Fund codes 511 – 767).
The contents of this report were developed under a grant from the U.S. Department of Education. However, these contents do not necessarily represent the policy of the U.S. Department of Education and you should not assume endorsement by the Federal Government.
The University of Massachusetts Donahue Institute is the public service, outreach, and economic development unit of the University of Massachusetts President’s Office. Established in 1971, the Institute strives to connect the Commonwealth with the resources of the University through services that combine theory and innovation with public and private sector applications.
The Institute’s Applied Research and Program Evaluation group specializes in applied social science research, including program evaluation, survey research, policy research, and needs assessment. The Research and Evaluation group has designed and implemented numerous innovative research and evaluation projects for a variety of programs and clients in the areas of education, human services, economic development, and organizational development.
University of Massachusetts Donahue Institute
Applied Research and Program Evaluation Group
100 Venture Way, Suite 5
Hadley, MA 01035-9462
(413) 587-2400
(413) 587-2410 to send a fax
www.donahue.umassp.edu
UMass Donahue Institute
Research and Evaluation Group
SRG Final Evaluation Report / Contents
Contents
Tables and Figures
Executive Summary i
Introduction i
Overview of SRG and Performance of the First Cohort of Level 4 Schools i
Key Study Findings iii
Introduction 1
Context and Overview of the Initiative 1
Methodology 3
I. Research Design 3
II. Schools Participating in the Study 4
All SRG Schools 4
The 2010 Decision Group 6
III. Data Sources 7
Documents 7
Interviews 8
ESE Consultant and Staff Interviews 8
District Leader Interviews 8
Site Visit Interviews 9
Targeted Interviews (Sustainability) 9
Surveys 9
Follow up – Spring 2014 Focal Inquiry (District Coaches and Partners) 10
IV. Analysis of Data 10
Qualitative Analysis 10
Longitudinal Analysis of Survey Data 11
Analysis of Exited and Non-Exited Schools 11
V. Limitations of the Study 12
Perspectives of All SRG Schools 13
I. Principal Perspectives on Overall Return of Specific Redesign Strategies 13
II. Perspectives on Key Elements of Practice 15
A. Curriculum and Instruction 16
Instructional Vision and Curriculum Alignment 16
Professional Development 16
Tiered Instruction 17
Instructional Coaches and Specialists 17
B. Leadership 18
Educators’ Perspectives of School Leadership Practices 18
Distributed Leadership 18
District Leadership and Support for Redesign 19
C. Structures for Collaboration 19
Common Planning Time 20
Restructuring the School Day 20
D. Data Use and Management 20
Principal and Leader Buy-in and Perspective on Value 20
Support with Data from Districts and ESE 21
E. School Culture and Climate 21
Professional and Student Culture 21
Redesign and Burnout 22
III. Discussion 22
Differences Between Exited and Not Exited Schools 25
I. Overview of Differences 26
II. Factors Associated with Improvement 26
A. Perceived Effectiveness and Return of Specific Turnaround Strategies 26
Implementing “Double-Yield” Strategies 28
Other Reflections on Turnaround Strategies 29
B. Differences in Relation to Key Elements of School Redesign 29
Curriculum and Instruction 29
Leadership Practices 34
District Support and Resources 40
Data Use and Management, and the Role of Data in Tiered Instruction 46
External Partnerships 48
School Culture and Climate 50
Summary of the Differences between Exited and Not Exited Schools 53
Sustainability of Redesign Progress 54
ESE Support for School Redesign 58
Conclusion 62
Appendix A: Principal Survey Instrument
Appendix B: Principal Survey Results
Appendix C: Educator Survey Instrument
Appendix D: Educator Survey Results
UMass Donahue InstituteApplied Research & Program Evaluation
SRG Final Evaluation Report / Tables and Figures
Tables and Figures
Table 1. Massachusetts SRG Schools 5
Table 2. Exit Recommendations for the SRG 2010 Decision Group 7
Table 3. All SRG Principal Surveys Completed 9
Table 4. All SRG Educator Surveys Completed 9
Table 5. Number of Surveys Received from 2010 Decision Cohort Groupings 12
Table 6. Principal Perceptions of Return on Investment by Strategy 14
Table 6 (continued). Principal Perceptions of Return on Investment by Strategy 15
Table 7. Principals’ Perceptions of AROI/Effectiveness of Redesign Strategies, by Group 27
Table 8. Double-Yield Strategies: Principals’ Perceptions 28
Table 9. Educators’ Perceptions of Curriculum and Instructional Practices 30
Table 10. Educators’ Perceptions of All PD 31
Table 11. Educators’ Perceptions of District-provided PD 32
Table 12. Educators’ Perspectives on their own Professional Growth 32
Table 13. Educators’ Perceptions of School Leadership Practices 35
Table 14. Educators’ Perceptions of School Leadership Direct Support 36
Table 15. Principals’ Perceptions of District Supports to School 41
Table 16. Educators’ Perceptions of District Supports 41
Table 17. District Supports to Principals 42
Table 18. Principals’ Perceptions of Autonomy from the District 43
Table 19. Educators’ Data Management and Use Practices 47
Table 20. Principals’ Perceptions of Barriers to Tiered Instruction 48
Table 21. Principal and Educator Perceptions of Partnerships’ Effectiveness Promoting Change 49
Table 22. Educators’ Sense of Being Overwhelmed 51
Table 23. Principals’ Perceived Effectiveness of Strategies in Promoting Change 52
Table 23 (Continued). Principals’ Perceived Effectiveness of Strategies in Promoting Change 53
Table 24. Principals’ Perceptions of Sustainability of Redesign Strategies by AROI Level 54
Table 24 (Continued). Principals’ Perceptions of Sustainability of Redesign Strategies by AROI Level 55
Table 25. Educators’ View of Sustainability 55
Table 26. Principals’ Perspectives of ESE Support: Resources and Tools 59
Table 27. Principals’ Perspectives of ESE Support: Processes 60
UMass Donahue InstituteApplied Research & Program Evaluation
SRG Final Evaluation Report / Executive Summary
Executive Summary
Introduction
The 2010 Act Relative to the Achievement Gap signified a renewed and expanded focus on improving student learning in traditionally underperforming (Level 4 and 5) schools in Massachusetts. Specifically, the Act provided districts greater power and authority with respect to those schools, and it designated districts as the official point of entry for the Massachusetts Department of Elementary and Secondary Education (ESE). Largely, the Act signified that districts would be held accountable for reversing the patterns of underperformance that had plagued their systems in the past. “Turnaround” was the new order of the day. Additionally in 2010, ESE established the Framework for District Accountability and Assistance. The Framework introduced a five-level scale to identify schools based on need and incorporated the Conditions for School Effectiveness, thereby linking accountability and assistance measures.
The Massachusetts School Redesign Grant (SRG) initiative is used by ESE as a key lever to support historically underperforming schools in their efforts to increase student achievement and build capacity to sustain improvements over time. Since 2010, five cohorts of schools have received competitive funding and technical support to develop and implement redesign plans, which, over a 3-year period, are intended to support schools in moving out of their patterns of chronic underperformance and yield gains in student learning.
In 2012, the Massachusetts Department of Elementary and Secondary Education (ESE) engaged the University of Massachusetts Donahue Institute (UMDI or the Institute) to design and conduct a program evaluation of the SRG initiative that would provide formative feedback to inform ESE’s management decisions and summative findings to assess whether the initiative is associated with positive outcomes. This evaluation study explores relationships between various elements of schools’ redesign processes and investigates why certain redesign strategies, and combinations of strategies, appear more likely than others to be associated with positive outcomes. The study addresses two primary research questions, as follows:
· Primary Research Question 1: How can ESE best support and manage the SRG project?
· Primary Research Question 2: In what ways and to what extent is SRG associated with trends in schools’ turnaround progress and improvements in student success?
Data sources included interviews with district and school leaders; online surveys of principals’ and educators’ beliefs, attitudes and perspectives on redesign; and document review. Through an analysis of educators’ and leaders’ perceptions of various aspects of their redesign experience, the study identifies those elements of redesign that are perceived to be associated with schools’ positive trajectories, highlights threats to the sustainability of improvement, identifies key ingredients of ESE’s turnaround support, and articulates implications for ESE’s future efforts to guide and strengthen struggling schools and build district capacity to support them.
Overview of SRG and Performance of the First Cohort of Level 4 Schools
In 2010, Massachusetts launched its first Request for Proposals inviting districts to apply for School Redesign Implementation Grant funding. These funds were made available initially through a combination of Massachusetts’ FY09 federal American Recovery and Reinvestment Act (ARRA and US Department of Education School Improvement Grants (SIG) funds. Having recently enacted policy reforms relevant to school improvement, Massachusetts endeavored to integrate the federal requirements into its emerging policy frameworks and positions. To do this, the state conceptualized SRG to reflect three overriding principles: a) Districts would be the point of entry for the Department. The theory was that the Department would partner with districts to develop their capacity to turn around their lowest performing schools, so that they could improve their current set of underperformers and also apply that knowledge to its next group of low performers, thereby improving performance across the system. This new theory of action also reflected an acknowledgment that historically the greatest number of underperforming schools tended to cluster in the same small set of districts. b) Only a limited number of schools would receive funding, so that the Department could provide substantial and meaningful support, and c) Reciprocal accountability: increases in accountability would be met with greater assistance from the Department.
In 2010, the Massachusetts’ accountability system identified 35 schools as Level 4 schools that were required by state law to turnaround in three years. Of these 35 schools, 31 successfully applied for and were granted SRG funding in either 2010, 2011, or 2012. In 2013, the ESE reviewed the performance trajectories of the 35 schools that were originally designated Level 4 status in 2010 and released recommendations about those schools’ accountability levels. These schools are referenced throughout this report as the 2010 Decision Group.
Of the 31 schools receiving SRG funds, regardless of whether they had been granted SRG funding for three, two or one year, over half of the schools (18 schools, or 58%) demonstrated improvement or evidence of a positive trajectory. Specifically:
· 13 SRG schools (42%) exited Level 4 status (moving up to Level 3, 2, or 1).
· 5 SRG schools (16%) were determined to be on a relatively promising trajectory but remained in Level 4 status for at least one more year.
· 10 SRG schools (32%) were determined to be on a flat or declining trajectory. In these cases, districts opted to partner with educational management operators, and the schools remained in Level 4 status.
· 3 schools (10%) were assigned Level 5 status, moving into state receivership.
Given a growing national recognition that turnaround requires time—as indicated by recent revisions to the federal guidelines expanding the timeline for turnaround grants from three to five years—these first round results may be considered promising.
Key Study Findings
Key findings, presented below, articulate the ways in which the initiative is associated with improvement across all the SRG schools and identifies elements of redesign that distinguish schools that have demonstrated improvement from schools that have not.
Finding 1 SRG represents a very promising and more effective model for catalyzing improvement in struggling Massachusetts schools.
Consistently, leaders and educators affirmed that the initiative signaled a departure from prior reform efforts, which were often characterized by multiple—and often misaligned—short-term strategies that yielded little success. By contrast, reports from the field suggest that SRG provides the tools and mechanisms necessary for schools and districts to think strategically about barriers to improvement in their specific contexts, and to institute processes to address them. More broadly, district and school leaders report that a more effective model of interaction between ESE and Massachusetts school districts emerged. The centerpiece of this model is a perceived shift by ESE from a focus on compliance to a more productive, assistance-focused approach that emphasizes partnership and collaboration.