Massachusetts Mathematics and Science Partnership Title IIB

Massachusetts Mathematics and Science Partnership Title IIB

Annual State-level Evaluation Report (Year 12)

Reporting Period: September 1, 2014 through August 31, 2015

Cumulative Reporting Period: February 2, 2004 through August 31, 2015

Prepared for the Massachusetts Department of Elementary and Secondary Education

April 2016

MMSP State-level Evaluation (Year 12) / Contents

Contents

Introduction 1

Program Overview 1

Evaluation Plan and Activities 2

Report Content and Organization 4

MMSP Program Activity 5

Overview of Partnerships 5

Overview of Courses 6

Course Participants 8

Educators’ Perceptions of MMSP Activities 19

Early Changes and Outcomes 22

Content Knowledge Gains 22

Changes in Instructional Practice and Student Growth Measures 23

Course Institutionalization 24

Recommendations and Strategic Considerations 26

Summary of Findings 29

Appendix A: Timeline for State-level Evaluation & TA Activities (Year 12) 32

Appendix B: Overview of MMSP Partnership Participation (All Funding Periods) 33

Appendix C: Overview of Budgets & Partnerships (All Funding Periods) 35

Appendix D: Enrollment and Attrition Information, by Course (Year 12) 38

Appendix E: Types of Schools and Districts (All Funding Periods) 39

Appendix F: Content Knowledge Pre- and Post-Assessment Scores,

by Course (Year 12) 40

/ UMass Donahue Institute
Research and Evaluation Group / II
MMSP State-level Evaluation (Year 12) / List of Tables

List of Tables

Table 1: Partnership Budgets (Year 12) 6

Table 2: Enrollment and Attrition Information, by Partnership (Year 12) 7

Table 3: Repeat Participation (Year 12) 8

Table 4: Participants, by School Type (All Funding Periods) 8

Table 5: High-Need District Status of Participants (All Funding Periods) 9

Table 6: Participants from High Need Districts, by Partnership (Year 12) 10

Table 7: Professional Position of Participants (Year 12) 12

Table 8: Teaching Content Areas of Participants (Year 12) 13

Table 9: Experience of Participants (Year 12) 14

Table 10: Teaching Level of Participants (Year 12) 14

Table 11: Licensure and Degrees Held – Regular Education Mathematics Teachers (Year 12) 15

Table 12: Licensure and Degrees Held – Special Education Mathematics Teachers (Year 12) 15

Table 13: Licensure and Degrees Held – Regular Education Science Teachers (Year 12) 16

Table 14: Licensure and Degree Held – Special Education Science Teachers (Year 12) 16

Table 15: Degrees Pursued (All Funding Periods) 17

Table 16: Reasons for Participation in MMSP Courses (All Funding Periods) 18

Table 17: Overall Ratings of MMSP Courses (Year 12) 19

Table 18: Perceptions of MMSP Course Delivery (Year 12) 20

Table 19: Perceptions regarding Alignment of MMSP Courses to Improvement Priorities (Year 12) 20

Table 20: Courses with Statistically Significant Gains (All Funding Periods) 23

Table 21: Perceptions about Student Growth Measures (Year 12) 24

/ UMass Donahue Institute
Research and Evaluation Group / II
MMSP State-level Evaluation (Year 12) / Introduction

Introduction

Program Overview

The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to improve student achievement in mathematics, science, and technology/engineering through intensive, high-quality professional development activities that focus on deepening teachers’ content knowledge. This multi-year project is funded by the U.S. Department of Education (USED) as part of the No Child Left Behind (NCLB) Act Title IIB funding stream. Funding to local partnerships is administered by state education agencies; in Massachusetts this is the Massachusetts Department of Elementary and Secondary Education (ESE), which awards funding through a competitive grant process.

Partnerships awarded these grants are required to include (1) a core partner that has been identified as a high-need school district in the subject matter on which the partnership is focusing and (2) a core partner that is a science, technology and/or engineering, or mathematics (STEM) department from an institution of higher education. The partnerships are composed of higher education institutions, school districts, and, in some cases, private organizations involved in providing both pre-service and in-service training to teachers. Partnership activities are guided by the following goals and objectives:

Goal I Develop and implement an effective and sustained course of study for in-service educators of STEM by integrating the courses of study into schools of arts and sciences and/or education at institutions of higher education.

Goal II Identify credible, instructionally useful measures of student growth and select, develop, and pilot assessments that measure student growth relative to subject matter standards.[1]

Goal III Increase the number of STEM teachers in the partner school districts who participate in effective professional development and advance their knowledge of subject matter standards, disciplinary practices, and student learning.

Goal IV Develop and implement a systemic approach to STEM education by integrating professional development with district and school STEM improvement initiatives.

All professional development funded by the program must be implemented in accordance with the Massachusetts Standards for Professional Development. Those standards encourage a set of coherent learning experiences that is systematic, purposeful, and structured over a sustained period of time with the goal of improving teacher practice and student outcomes. Each course offered by the partnerships must provide at least 45 hours of direct instruction followed by 24 hours[2] of supplemental activities to guide the implementation of standards-based instruction and facilitate connections between the course(s) and student growth measures. Partnerships are encouraged to tailor the model used to deliver the professional development and follow-up to best fit the objectives, resources, expertise, and existing infrastructure of their programs. At least half of participating educators must be from high-need districts.

The program began in February 2004 and has had 12 funding periods, defined as follows:

·  Year 1: February 2, 2004 through August 31, 2004 (initial funding for Cohort 1)

·  Year 2: September 1, 2004 through August 31, 2005 (initial funding for Cohort 2)

·  Year 3: September 1, 2005 through August 31, 2006

·  Year 4: September 1, 2006 through August 31, 2007 (initial funding for Cohort 3)

·  Year 5: September 1, 2007 through August 31, 2008

·  Year 6: September 1, 2008 through August 31, 2009 (initial funding for Cohort 4)

·  Year 7: September 1, 2009 through August 31, 2010

·  Year 8: September 1, 2010 through August 31, 2011 (initial funding for Cohort 5)

·  Year 9: September 1, 2011 through August 31, 2012 (initial funding for Cohort 6)

·  Year 10: September 1, 2012 through August 31, 2013

·  Year 11: September 1, 2013 through August 31, 2014 (initial funding for Cohort 7)

·  Year 12: September 1, 2014 through August 31, 2015 (initial funding for Cohort 8)

Evaluation Plan and Activities

The UMass Donahue Institute (UMDI) has served as the state-level evaluator of MMSP since the program’s inception. Beginning with Cohort 7, ESE selected UMDI to conduct both statewide and local evaluation research activities for the program.

The statewide and local evaluations are guided by the following evaluation goals:

1. Facilitate partnership’s annual federal reporting and provide ESE with a statewide report of the data required for federal reporting.
2. Provide ESE and partnerships with timely formative feedback to support program implementation and refinement.
3. Assess statewide and partnership progress toward achievement of the program’s articulated goals.
4. Track changes in the achievement of students taught by MMSP participants.

Data collection for the evaluation is organized around a basic logic model for professional development initiatives, as shown below.

The statewide and local evaluations use a common approach and structure drawing on the data sources described below.

End-of-course report packages

Over the years, UMDI and ESE established an ongoing reporting system for all partnerships, consisting primarily of end-of-course summary report packages. Those report packages include contact information of course participants; course enrollment and completion rates; individual pre/post-results of content knowledge assessments; and completed course participant surveys.[3] End-of-course packages gather data relative to multiple evaluation goals, namely federal reporting (goal 1), formative feedback (goal 2), and assessment of progress toward program goals (goal 3). In the past, the end-of-course surveys focused primarily on obtaining background information about participants. In consultation with ESE program staff, the survey was streamlined in 2014 to be more feedback oriented with the intent of pilot testing the use of ESE data to help supplement data collected from end-of-course packages. UMDI provides partnerships with reports of survey results for each MMSP course.

ESE datasets

In 2015, UMDI pilot-tested the use of ESE datasets to provide information to partnerships and ESE about MMSP participants for federal reporting. These data sets include: (1) EPIMS teacher-level data for work assignment and demographic information about teachers and administrators participating in MMSP courses; and (2) Student Course Schedule (SCS) student-level data for a count of the number of students taught by participating teachers. Additionally, ELAR (Educator Licensure and Renewal) data were used to describe the types of licenses and certifications held by MMSP teachers for this annual statewide report.

Partnership Interviews

Phone interviews with the core partners of each grant focus on gathering data relative to evaluation goals 2 (timely formative feedback) and 3 (progress toward meeting the program goals). Other significant partners are added at the discretion of the evaluation team with input from the partnership and ESE. UMDI staff conducted interviews and provided management briefs (Spring 2015) to each partnership and ESE.

Annual Participant Survey

In addition to end-of-course surveys, UMDI developed and administered an annual survey of MMSP course participants in the early summer of 2015. This survey provides an opportunity to gather participant feedback on the partnership’s professional development program more broadly, including the follow-up component and how the individual course offerings fit together into a coherent course of study. Participants are also able to reflect upon the extent to which their participation affected their classroom practice.

Student Growth Measures

UMDI will work with each partnership to fully understand the student growth measures being used and develop a process through which those partnerships can provide data for analysis in relation to evaluation goal 4 (tracking changes in student achievement). This will result in analysis reports for each partnership at the end of the grant as well as a statewide summary report for ESE.

A detailed timeline of evaluation and technical assistance activities carried out in Year 12, can be found in Appendix A.

Report Content and Organization

This report summarizes findings from data collection efforts conducted during Year 12. It is organized into four main sections, covering the following topics:

1.  MMSP Program Activity provides an overview of MMSP partnerships and participant characteristics for funding Year 12 (2014–15) combined with cumulative data since the program’s inception.

2.  Educator’s Perception of MMSP Activities focuses on the quality, utility, and relevance of professional development provided in Year 12.

3.  Early Changes and Outcomes describes reported changes in content knowledge, instructional practices, and the institutionalization of courses.

4.  Recommendations and Strategic Considerations presents the perspectives of educators participating in MMSP courses and partnership feedback regarding implementation and ESE support.

5.  Summary of Findings lists the key findings, for both Year 12 and overall.

/ UMass Donahue Institute
Research and Evaluation Group / 18
MMSP State-level Evaluation (Year 12) / Program Activity

MMSP Program Activity

This section offers information and findings regarding MMSP program activity for funding Year 12 along with cumulative data reflecting activity since the inception of the MMSP program in 2004. More specifically, it presents an overview of partnership awards and funding, followed by an overview of courses, and finally information about target participants reached.

A number of data sources were analyzed to compile Year 12 data. The primary source was the end-of-course packages, including enrollment reports and course participant listings. Sources such as ESE School and District Profile data and end-of-year SY15 ESE individual-level datasets (EPIMS15D, SCS15D, ELAR) were also used. Cumulative findings are derived from the MMSP Annual State-Level Evaluation Report (Year 11),[4] submitted to ESE in January 2015, and cumulative files of survey data from the previous 11 years. Comparisons of cumulative and Year 12 findings are presented in cases were data sources are the same or, at a minimum, are comparable.

Overview of Partnerships

The MMSP program awarded more than $20.7 million[5] to a total of 45 partnerships (comprising 8 cohorts) since its inception in 2004. Of these, 23 were organized around mathematics content, 17 were organized around science content, and 5 were organized around both mathematics and science content. It should be noted that while some partnerships were awarded funding in more than one funding period, for evaluation purposes, a partnership is counted in this total each time it received funding for a new grant. For an overview of MMSP partnership participation see Appendix B, and for more detailed information about budgets for all partnerships and all funding periods see Appendix C.

In Year 12 (2014–15), funding was awarded to 13 MMSP partnerships.[6] This includes six new partnerships (Cohort 8) that completed their first year of work, six Cohort 7 partnerships that completed their second year, and one Cohort 6 partnership that received an extension. All but one of these partnerships offered both content-specific training (in science, mathematics, or both areas) and supplemental support in Year 12 to ensure classroom implementation. One partnership, Worcester, did not offer direct instruction in Year 12, but did offer follow-up support to teachers for a course offered during the previous grant year. Through a grant extension—which provided some funds beyond the typical three-year period—one Cohort 6 partnership (Northeastern) created an opportunity for a cohort of teachers to share work products, lessons learned, and strategies for maintaining student gains. Table 1 shows the funding received by each partnership in Year 12 as well as the content areas targeted by the partnerships’ professional development efforts.

Table 1: Partnership Budgets (Year 12)
Cohort & PD Content Area(s) / Partnership / Sep14–Aug15
Cohort 6 – Science / Northeastern / $14,998
Cohort 7 – Math/Science / Fitchburg / $71,404
Cohort 7 – Science / Framingham / $126,141
Cohort 7 – STE / Global Learning Public Charter / $109,096
Cohort 7 – Math / Lesley – Springfield-Easthampton / $310,943
Cohort 7 – Science / West Springfield Public Schools / $50,003
Cohort 7 – Math / Worcester Public Schools / $62,636
Cohort 8 – Math / Billerica / $207,800
Cohort 8 – Math/Science / Fall River / $70,869
Cohort 8 – Math/Science / Lesley – Brockton-Weymouth / $201,822
Cohort 8 – Science / Lowell / $45,045
Cohort 8 – Science / Methuen / $68,195
Cohort 8 – Science / UMASS Boston 5DP / $40,522
Total / $1,379,474
Source: Budget information from Annual Performance Reports (FY14–15). Professional development (PD) content area from worksheets provided by ESE.

Overview of Courses

Course Enrollment and Attrition

Overall, 455 courses were delivered throughout the course of the MMSP program. Of these courses, 304 (67%) focused on mathematics, 149 (33%) focused on science, and 2 (less than 1%) were courses offering both mathematics and science/technology content. Of the courses offered, 255 (56%) were delivered once, and 200 (44%) were repeat offerings.