Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Cohort 4 Reporting Period: September 1, 2008, through August 31, 2010
Cumulative Reporting Period: February 2, 2004, through August 31, 2010
Prepared for the Massachusetts Department of Elementary and Secondary Education
June 2011
Annual Report of the MMSP State-level Evaluation / ContentsContents
Program Description
Report Organization
Evaluation Plan and Activities
State-level Evaluation
Local Evaluation and Related Technical Assistance
Cohort 4 Activity: September 1, 2008 through August 31, 2010
State-level Participant Background Data
Partnership-level Participant Background Data
Cumulative Summary: All Cohorts over All Funding Periods
Summary of Findings
Cumulative Findings
Cohort 4 Findings
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Tables
Tables Index
Table 1:Budgets: Cohort 4 Partnerships
Table 2:Teaching Areas: Cohort 4 Participants
Table 3:Types of Schools of Unique Participants: Cohort 4 Participants
Table 4:High Need Status of Unique Participants from Public Schools: Cohort 4 Participants
Table 5:High Need District Participants by Partnership: Cohort 4 Participants
Table 6:Reasons for Participation: Cohort 4, All Seats
Table 7:Repeat Participants: Cohort 4 Partnerships
Table 8:Total Enrollment and Attrition Information: Cohort 4 Partnerships
Table 9a:Cohort 4 Science and Technology Teaching Areas – Regular Education
Table 9b:Cohort 4 Science and Technology Teaching Areas – Special Education
Table 9c:Cohort 4 Science and Technology Teaching Areas – ELL Education
Table 10a:Cohort 4 Mathematics Teacher Levels – Regular Education
Table 10b:Cohort 4 Mathematics Teacher Levels – Special Education
Table 10c:Cohort 4 Mathematics Teacher Levels – ELL Education
Table 11:Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores
for Cohort 4
Table 12:Boston Public Schools Participant Background Information (M)
Table 13:Brockton Public Schools Participant Background Information (M)
Table 14:Gateway Regional School District Participant Background Information (S)
Table 15:Lesley University C4 Participant Background Information (M)
Table 16:Greater North Shore Participant Background Information (S)
Table 17:Randolph Public Schools Participant Background Information (S)
Table 18:Springfield College Participant Background Information (S)
Table 19:Boston University Participant Background Information (M)
Table 20:Overview of MMSP Partnership Participation
Table 21:Budgets: All Partnerships, All Funding Periods
Table 22:Teaching Areas: All Participants, All Funding Periods
Table 23.Types of Schools: All Unique Participants, All Funding Periods
Table 24:High Need Status of All Unique Participants from Public Schools
Table 25:Reasons for Participation of Participants: All Seats, All Funding Periods
Table 26:Repeat Participants: All Partnerships, All Funding Periods
Table 27:Highly Qualified Status: All Unique Participants, All Funding Periods
Table 28:MTEL Tests Taken by All Participants – Total to Date
Table 29a:Science and Tech/Engineering Teaching Areas of All Participants – Regular Education
Table 29b:Science and Tech/Engineering Teaching Areas of All Participants – Special Education
Table 29c:Science and Tech/Engineering Teaching Areas of All Participants – ELL Education
Table 30a:Mathematics Teacher Levels of All Participants – Regular Education
Table 30b:Mathematics Teacher Levels of All Participants – Special Education
Table 30c:Mathematics Teacher Levels of All Participants – ELL Education
Table 31:Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Program Description
Program Description
The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to improve student achievement in mathematics, science, and technology/engineering through intensive, high-quality professional development activities that focus on deepening teachers’ content knowledge. This multi-year project is funded by the U.S. Department of Education as part of the No Child Left Behind (NCLB) Act Title IIB funding stream. Funding to local partnerships is administered by state education agencies; in Massachusetts this is the Massachusetts Department of Elementary and Secondary Education (ESE), which awards funding through a competitive grant process.
Partnerships awarded these grants are required to include 1) a core partner that has been identified as a high need school district in the subject matter on which the partnership is focusing and 2) a core partner that is a science, technology and/or engineering, or mathematics (STEM) department from an institution of higher education. The partnerships are composed of higher education institutions, school districts, and, in some cases, private organizations involved in providing both pre-service and in-service training to teachers. Partnerships are required to offer courses that equal at least 45 hours of direct instruction followed by at least 20 hours of follow-up contact to support the implementation of course content in the classroom. Partnerships are encouraged to tailor the model used to deliver the professional development and follow-up to best fit the objectives of their programs along with their resources, expertise, and existing infrastructure. Partnership activities are guided by the following goals[1]:
Goal I Develop and implement an effective and sustained course of study for in-service teachers of STEM by integrating the courses of study into schools of arts and sciences and/or education at institutions of higher education.
Goal IIIncrease the number of STEM teachers in the partner school districts who are licensed in the subject area(s) and grade level(s) they teach.
Goal IIIIncrease the number of STEM teachers in the partner school districts who participate in high quality professional development and advance their content knowledge.
Goal IVDevelop and implement a systemic approach to STEM education by integrating professional development with district and school STEM improvement initiatives.
The program began in February 2004, and has had seven funding periods, defined as follows:
- Year 1: February 2, 2004, through August 31, 2004
- Year 2: September 1, 2004, through August 31, 2005
- Year 3: September 1, 2005, through August 31, 2006
- Year 4: September 1, 2006, through August 31, 2007
- Year 5: September 1, 2007, through August 31, 2008
- Year 6: September 1, 2008, through August 31, 2009
- Year 7: September 1, 2009, through August 31, 2010.
The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial funding in Year 2 are referred to as Cohort 2; those who received initial funding in Year 4 are referred to as Cohort 3; and those who received initial funding in Year 6 are referred to as Cohort 4.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Report Organization
Report Organization
The purpose of this report is twofold: 1) to provide details regarding only Cohort 4 participation for the two years of funding for it thus far and2) to provide a cumulative summary regarding participation for all cohorts over all funding periods.
Data supporting the first purpose address the period of September 1, 2008, through August 31, 2010; data supporting the second purpose address the period of February 2, 2004, through August 31, 2010. Participant data were collected through the Participant Background Survey, an instrument developed by the UMass Donahue Institute (UMDI) and administered by partnerships to each participant on the last day of each course. See Appendix A for the survey used during Year 7. The purpose of this survey is to gather data about participants’ professional backgrounds and qualifications. This information provides a picture of who the participants are, aids in determining whether the courses are reaching the teachers who most need professional development, and aids in tracking how teacher qualifications may change during the MMSP funding period. Data from the survey regarding teacher licensure, possession of and progress towards earning degrees, and status in terms of Massachusetts Tests for Educator Licensure (MTEL) exams allows determination of the number of teachers who meet criteria defining highly qualified status. Unless noted, data from the survey are reported in terms of unique individuals, regardless of the number of courses taken by each individual.
Data speaking to the strengthening of relationships between partnership members were collected through a section of the local evaluation reports that partnerships were required to submit to the ESE. In this section, partnerships were asked to describe the extent to which their courses had been integrated into activities of their higher education partners.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Evaluation Plan and Activities
Evaluation Plan and Activities
State-level Evaluation
Although not required by the U.S. Department of Education, the ESE contracted with UMDI to conduct a state-level evaluation of the MMSP. UMDI’s primary role as state-level evaluator is to coordinate program-wide collection of outcome data on behalf of the ESE. Data collection for the state-level evaluation is organized around a basic logic model for professional development initiatives shown below.
Local Evaluation and Related Technical Assistance
In addition to the state-level data collection, each partnership is required to conduct its own local evaluation. In an effort to support strong local evaluations, ESE required that partnerships sub-contract with UMDI to provide technical assistance on design and implementation of their local evaluations. The timeline listing the evaluation activities is found in Appendix B.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Cohort 4 Activity
Cohort 4 Activity: September 1, 2008 through August 31, 2010
Cohort 4, which began in the 2008-2009 funding period, consisted of eight partnerships. Table 1 shows the funding received by Cohort 4 partnerships for the two periods between September 2008 and August 2010.
Table 1: Budgets: Cohort 4 PartnershipsPartnership / Sep08-Aug09 / Sep09-Aug10 / TOTAL
Boston Public Schools (M) / $157,975 / $405,747 / $563,722
Brockton Public Schools (M) / $180,145 / $255,758 / $435,903
Gateway Regional School Dist. (S) / $186,609 / $200,370 / $386,979
Lesley Springfield (M) / $228,593 / $324,820 / $553,413
Greater NorthShore (S) / $265,917 / $306,690 / $572,607
Randolph Public Schools (S) / $176,993 / $183,150 / $360,143
SpringfieldCollege (S) / $161,062 / $148,896 / $309,958
BostonUniversity (M) / $241,586 / $245,180 / $486,766
TOTAL / $1,598,880 / $2,070,611 / $3,669,491
State-level Participant Background Data
Cohort 4 consisted of eight partnerships, with four of the eight partnerships offering mathematics professional development and four offering science professional development. There were 74 Cohort 4 courses delivered. Of these 74 courses, 46 were mathematics courses and 28 were science courses. Of those 74 courses, 33 (45%) were unique, and 41 (55%) were repeat offerings. This section of the report summarizes data collected from participants in these courses.
By the end of the 2009-2010 funding period for Cohort 4,840 unique participants completed the Participant Background Survey on one or more occasions.Of these 840 participants,233 took two or more courses within Cohort 4, which translated to 1,230 course seats. The term “unique participant” refers to each individual who participated in the program, regardless of how many courses he or she took. Data for items from the survey that help to convey participants’ professional backgrounds and motives for participation are discussed in the remainder of this section. All survey data for the group completing the survey in 2009-2010 may be found in Appendix C. The responses to the survey questions are presented as frequencies and percentages. Not all percentages total 100% because many items allowed multiple responses and not all of the participants responded to all of the items.
Position of Participants
At the time of their last completed survey from a Cohort 4 MMSP course, 96% of course participants identified themselves as teachers. Of all respondents, 72% were regular education teachers; 19% were special education or special education inclusion teachers; 5% were ELL teachers; <1% were long-term substitutes; <1% were paraprofessionals; <1% were principals, assistant principals, or headmasters; and 3% indicated that they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 2. Because respondents identified all teaching areas that applied to their positions at the time of the survey, some selected multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 38% of Cohort 4 participants were teaching mathematics (including those teaching mathematics at the elementary level),34% were teaching science, and 31% were teaching all subjects at the elementary level.
Table2: Teaching Areas: Cohort 4 ParticipantsTeaching Areas
(Multiple responses permitted) / Sep08-Aug09
N =341 / Sep09-Aug10
N =630 / TOTAL
N =840
n / % / n / % / n / %
Mathematics / 102 / 30 / 184 / 29 / 250 / 30
Any science area / 138 / 41 / 213 / 34 / 286 / 34
General Science / 98 / 29 / 123 / 20 / 175 / 21
Biology / 27 / 8 / 52 / 8 / 67 / 8
Earth Science / 22 / 7 / 25 / 4 / 39 / 5
Chemistry / 17 / 5 / 31 / 5 / 43 / 5
Physics / 17 / 5 / 34 / 5 / 45 / 5
Technology/Engineering / 14 / 4 / 13 / 2 / 22 / 3
Elementary (all subjects) / 87 / 26 / 197 / 31 / 260 / 31
Elementary Mathematics / 21 / 6 / 55 / 9 / 67 / 8
Other / 11 / 3 / 26 / 4 / 37 / 4
Not Currently Teaching / 7 / 2 / 12 / 2 / 17 / 2
Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 840 unique Cohort 4 participants was as follows: 15% were in their first to third year of teaching, 41% had between four and ten years experience in education, 27% had between 11 and 20 years of experience, 14% reported over 20 years of experience, and 3% did not report.
Teaching Levels of Participants
For reporting purposes, schools in the participating districts were organized into categories of elementary schools (grades K-5), K-8 schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last completed survey from an MMSP course, 36% of Cohort 4 participants were teaching in an elementary or K-8 school, 44% were teaching in a middle school, 17% were teaching in a high school, and less than 1% were teaching at the pre-K level. Those remaining either were not currently teaching or the level at which they taught was unknown.
Types of Schools of Participants
As shown in Table 3, approximately98% of unique Cohort 4 participants worked in a public school setting, and approximately 3% worked in a non-public school setting.
Table 3: Types of Schools of Unique Participants: Cohort 4 ParticipantsSchool Type / Sep08–Aug09 / Sep09–Aug10 / TOTAL
n / % / n / % / n / %
Public Schools (includes public charter schools) / 333 / 98 / 614 / 98 / 819 / 98
Non-publicSchool / 7 / 2 / 16 / 3 / 21 / 3
Other or No Response / 1 / <1 / 0 / 0 / 0 / 0
TOTAL / 341 / 100 / 630 / 100 / 840 / 100
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for the high need designation.
The ESE expected that at least 50% of participants in each partnership would come from high need districts, and further, they set an informal goal that at least 75% of participants for each partnership would come from high need districts. Of all Cohort 4 participants, including those from non-public schools, 65% came from high need districts. Table 4 shows that by the end of the 2009-2010 funding period, 67% of Cohort 4 participants from public schools had come from high need districts.
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 4 ParticipantsSchool Type / Sep08–Aug09 / Sep09–Aug10 / TOTAL
n / % / n / % / n / %
High Need District / 213 / 64 / 412 / 67 / 545 / 67
Non-high Need District / 120 / 36 / 202 / 33 / 274 / 34
TOTAL / 333 / 100 / 614 / 100 / 819 / 101
Table 5 presents the number of participants from high need districts organized by each partnership. An examination of high need district participation in individual partnerships reveals that over the course of each partnership’s involvement in MMSP, three of the eight partnerships had at least 50% of their participants coming from high need districts and those same three also exceeded the informal goal of having at least 75% of the participants come from high need districts.
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in the earlier portions of this report where data were presented for unique participants, data for this topic are presented for all course seats, since responses are uniquely relevant to each distinct course that a participant took. Table 6 presents findings for all 1,230 seats for all courses taken by Cohort 4 participants through the end of the 2009-2010 funding period.
Table 5: High Need District Participants by Partnership: Cohort 4 ParticipantsPartnership / High Need Districts / Sep08–Aug09 / Sep09–Aug10 / TOTAL
n / % / n / % / n* / %
Boston PS (M) / Boston / 40 / 100% / 174 / 100% / 195 / 100%
Brockton PS (M) / Brockton / 21 / 28 / 49
Fall River / 9 / 12 / 21
Falmouth / 0 / 1 / 1
Plymouth / 0 / 3 / 3
Quincy / 0 / 2 / 2
Seekonk / 0 / 1 / 1
Swansea / 0 / 1 / 1
Freetown/Lakeville / 0 / 3 / 3
Subtotal / 30 / 59% / 51 / 45% / 81 / 49%
Gateway RSD (S) / Easthampton / 1 / 3 / 4
Gateway / 6 / 7 / 11
Holyoke / 2 / 9 / 11
Springfield / 8 / 0 / 2
Subtotal / 17 / 39% / 19 / 38% / 28 / 41%
Lesley Springfield (M) / Chicopee / 1 / 1 / 1
Holyoke / 9 / 6 / 10
Springfield / 35 / 59 / 70
Westfield / 4 / 5 / 5
Subtotal / 49 / 94% / 71 / 84% / 86 / 85%
Greater NorthShore (S) / Boston / 26 / 26 / 46
Lynn / 3 / 4 / 6
Malden / 0 / 1 / 1
PioneerCharterSchool of Science / 1 / 0 / 1
Randolph / 1 / 1 / 1
Revere / 0 / 8 / 8
Somerville / 0 / 12 / 12
Subtotal / 31 / 50% / 52 / 46% / 75 / 49%
Randolph PS (S) / Randolph / 6 / 25% / 18 / 41% / 19 / 34%
Springfield Coll. (S) / Springfield / 26 / 96% / 30 / 100% / 43 / 100%
BostonUniversity (M) / Boston / 3 / 3 / 6
Cambridge / 0 / 1 / 1
Falmouth / 2 / 0 / 2
Haverhill / 2 / 0 / 2
Lawrence / 7 / 1 / 8
Lowell / 0 / 2 / 2
Salem / 1 / 1 / 2
Somerville / 0 / 1 / 1
Woburn / 0 / 1 / 1
Worcester / 1 / 0 / 1
Subtotal / 16 / 39% / 10 / 29% / 26 / 38%
*Note that if this column is summed, the total will not correspond to data in Table 4 because 1) Table 4 presents data for unique participants in public schools across all partnerships while Table 5 presents data for all unique participants within partnerships (and some participants took courses across partnerships) and 2) The districts of some of those participants who crossed partnerships were not consistently classified as high need districts because ofcourse content.
Table 6: Reasons for Participation: Cohort 4, All SeatsReasons for Participation (Multiple responses permitted) / Sep08-Aug09 / Sep09-Aug10 / TOTAL
n / % of 391 course seats / n / % of839 course seats / n / %
of 1,230 course seats
To increase knowledge in content / 287 / 73% / 603 / 72% / 890 / 72%
To obtain graduate credit / 240 / 61% / 565 / 67% / 805 / 65%
To pursue a personal interest / 118 / 30% / 239 / 29% / 357 / 29%
To earn PDPs for recertification / 108 / 28% / 217 / 26% / 325 / 26%
To get an additional license (certification) / 52 / 13% / 123 / 15% / 175 / 14%
To follow an administrator’s suggestion / 30 / 8% / 105 / 13% / 135 / 11%
To prepare for the Massachusetts Test for Educator Licensure (MTEL) / 45 / 12% / 74 / 9% / 119 / 10%
To obtain a first license (certification) / 11 / 3% / 29 / 4% / 40 / 3%
Other / 28 / 7% / 37 / 4% / 65 / 5%
Repeat Participation
Cohort 4 partnerships were successful at encouraging and retaining repeat participants. All eight partnerships offered multiple courses, and all partnerships had participants who attended more than one course within Cohort 4. Of all unique Cohort 4 participants, 233 attended multiple courseswithin Cohort 4, which translated to 1,230 course seats filled for this cohort.Table 7 provides details regarding repeat participation, including information on the 65 repeat participants who took courses from partnerships from previous MMSP cohorts. (Table 12 through Table 19 also include data on repeat participation, as does Table 26.)
Table 7: Repeat Participants: Cohort 4 PartnershipsPartnership / Number of Courses Offered to Date / Total Number of Unique* Participants to Date / Number Taking Multiple Courses in Cohort 4 / Number Taking One Course in Cohort 4 & One or More Courses in Previous Cohorts / Number Taking Multiple Courses in Cohort 4 & Previous Cohorts
Boston Public Schools (M) / 16 / 192 / 48 / 8 / 56
Brockton Public Schools (M) / 9 / 163 / 3 / 22 / 25
Gateway Regional Sch Dis (S) / 2 / 66 / 27 / 5 / 32
Lesley Springfield (S) / 15 / 101 / 57 / 4 / 61
Greater NorthShore (S) / 19 / 150 / 60 / 15 / 75
Randolph Public Schools (S) / 5 / 55 / 17 / 5 / 22
SpringfieldCollege (S) / 4 / 44 / 14 / 3 / 17
BostonUniversity (M) / 4 / 69 / 7 / 3 / 10
TOTAL / 74 / 840 / 233 / 65 / 298
*Participants who participated in multiple courses across partnerships were counted only once in the partnership of their most recent course.