Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Cohort 3 Reporting Period: September 1, 2006, through August 31, 2009
Cohort 4 Reporting Period: September 1, 2008, through August 31, 2009
Cumulative Reporting Period: February 2, 2004, through August 31, 2009
Prepared for the Massachusetts Department of Elementary and Secondary Education
December 2010
Annual Report of the MMSP State-level Evaluation / ContentsContents
Program Description
Report Organization
Evaluation Plan and Activities
State-level Evaluation
Local Evaluation and Related Technical Assistance
Cohort 3 Activity: September 1, 2006 through August 31, 2009
State-level Participant Background Data
Partnership-level Participant Background Data
Cohort 4 Activity: September 1, 2008 through August 31, 2009
State-level Participant Background Data
Partnership-level Participant Background Data
Cumulative Summary: All Cohorts over All Funding Periods
Summary of Findings
Cumulative Findings
Cohort 3 Findings
Cohort 4 Findings
Appendix A: Participant Background Survey – September 2008 – August 2009
Appendix B:Timeline for State-level Evaluation and TA Activities
Appendix C: Results of the Participant Background Survey for 2008-2009
Appendix D:High Need District Eligibility Criteria
Appendix E: Enrollment and Attrition Rates by Course
Appendix F: Subject Matter Competency Demonstration Options
Appendix G: Mean Percentage Scores for Pre- & Post-course Tests
Appendix H: High Need Districts for All Funding Periods, by Partnership
Appendix I: Criteria that Account for Gain in Highly Qualified Status
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Tables
Tables Index
Table 1: Budgets: Cohort 3 Partnerships
Table 2: Teaching Areas: Cohort 3 Participants
Table 3: Types of Schools of Unique Participants: Cohort 3 Participants
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 3 Participants
Table 5: High Need District Participants by Partnership: Cohort 3 Participants
Table 6: Reasons for Participation: Cohort 3, All Seats
Table 7: Repeat Participants: Cohort 3 Partnerships
Table 8: Total Enrollment and Attrition Information: Cohort 3 Partnerships
Table 9a: Cohort 3 Science and Technology Teaching Areas – Regular Education
Table 9b: Cohort 3 Science and Technology Teaching Areas – Special Education
Table 9c: Cohort 3 Science and Technology Teaching Areas – ELL Education
Table 10a: Cohort 3 Mathematics Teacher Levels – Regular Education
Table 10b: Cohort 3 Mathematics Teacher Levels – Special Education
Table 10c: Cohort 3 Mathematics Teacher Levels – ELL Education
Table 11: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores:
Cohort 3
Table 12: EduTron Lowell Participant Background Information (M/S)
Table 13: EduTron Fitchburg Participant Background Information (M)
Table 14: Lesley University Participant Background Information (M)
Table 15: North Shore Participant Background Information (S)
Table 16: UMass Amherst Participant Background Information (M/S)
Table 17: Salem State College Participant Background Information (M)
Table 18: SE/Cape Participant Background Information (S)
Table 19: WPI Participant Background Information (S)
Table 20: Worcester Public Schools Participant Background Information (M)
Table 21: Budgets: Cohort 4 Partnerships
Table 22: Teaching Areas: Cohort 4 Participants
Table 23: Types of Schools of Unique Participants: Cohort 4 Participants
Table 24: High Need Status of Unique Participants from Public Schools: Cohort 4 Participants
Table 25: High Need District Participants by Partnership: Cohort 4 Participants
Table 26: Reasons for Participation: Cohort 4, All Seats
Table 27: Repeat Participants: Cohort 4 Partnerships
Table 28: Total Enrollment and Attrition Information: Cohort 4 Partnerships
Table 29a: Cohort 4 Science and Technology Teaching Areas – Regular Education
Table 29b: Cohort 4 Science and Technology Teaching Areas – Special Education
Table 29c: Cohort 4 Science and Technology Teaching Areas – ELL Education
Table 30a: Cohort 4 Mathematics Teacher Levels – Regular Education
Table 30b: Cohort 4 Mathematics Teacher Levels – Special Education
Table 30c: Cohort 4 Mathematics Teacher Levels – ELL Education
Table 31: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores for
Cohort 4
Table 32: Boston Public Schools Participant Background Information (M)
Table 33: Brockton Public Schools Participant Background Information (M)
Table 34: Gateway Regional School District Participant Background Information (S)
Table 35: Lesley University C4 Participant Background Information (M)
Table 36: Greater North Shore Participant Background Information (S)
Table 37: Randolph Public Schools Participant Background Information (S)
Table 38: Springfield College Participant Background Information (S)
Table 39: Boston University Participant Background Information (M)
Table 40: Overview of MMSP Partnership Participation
Table 41: Budgets: All Partnerships, All Funding Periods
Table 42: Teaching Areas: All Participants, All Funding Periods
Table 43. Types of Schools: All Unique Participants, All Funding Periods
Table 44: High Need Status of All Unique Participants from Public Schools
Table 45: Reasons for Participation of Participants: All Seats, All Funding Periods
Table 46: Repeat Participants: All Partnerships, All Funding Periods
Table 47: Highly Qualified Status: All Unique Participants, All Funding Periods
Table 48: MTEL Tests Taken by All Participants – Total to Date
Table 49a: Science and Tech/Engineering Teaching Areas of All Participants – Regular Education
Table 49b: Science and Tech/Engineering Teaching Areas of All Participants – Special Education
Table 49c: Science and Tech/Engineering Teaching Areas of All Participants – ELL Education
Table 50a: Mathematics Teacher Levels of All Participants – Regular Education
Table 50b: Mathematics Teacher Levels of All Participants – Special Education
Table 50c: Mathematics Teacher Levels of All Participants – ELL Education
Table 51: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Program Description
Program Description
The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to improve student achievement in mathematics, science, and technology/engineering through intensive, high-quality professional development activities that focus on deepening teachers’ content knowledge. This multi-year project is funded by the U.S. Department of Education as part of the No Child Left Behind (NCLB) Act Title IIB funding stream. Funding to local partnerships is administered by state education agencies; in Massachusetts this is the Massachusetts Department of Elementary and Secondary Education (ESE), which awards funding through a competitive grant process.
Partnerships awarded these grants are required to include 1) a core partner that has been identified as a high need school district in the subject matter on which the partnership is focusing and 2) a core partner that is a science, technology and/or engineering, or mathematics (STEM) department from an institution of higher education. The partnerships are composed of higher education institutions, school districts, and, in some cases, private organizations involved in providing both pre-service and in-service training to teachers. Partnerships are required to offer courses that equal at least 45 hours of direct instruction followed by at least 20 hours of follow-up contact to support the implementation of course content in the classroom. Partnerships are encouraged to tailor the model used to deliver the professional development and follow-up to best fit the objectives of their programs along with their resources, expertise, and existing infrastructure. Partnership activities are guided by the following goals[1]:
Goal I Develop and implement an effective and sustained course of study for in-service teachers of STEM by integrating the courses of study into schools of arts and sciences and/or education at institutions of higher education.
Goal IIIncrease the number of STEM teachers in the partner school districts who are licensed in the subject area(s) and grade level(s) they teach.
Goal IIIIncrease the number of STEM teachers in the partner school districts who participate in high quality professional development and advance their content knowledge.
Goal IVDevelop and implement a systemic approach to STEM education by integrating professional development with district and school STEM improvement initiatives.
The program began in February 2004, and has had six funding periods, defined as follows:
- Year 1: February 2, 2004 through August 31, 2004
- Year 2: September 1, 2004 through August 31, 2005
- Year 3: September 1, 2005 through August 31, 2006
- Year 4: September 1, 2006 through August 31, 2007
- Year 5: September 1, 2007 through August 31, 2008
- Year 6: September 1, 2008 through August 31, 2009.
The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial funding in Year 2 are referred to as Cohort 2; those who received initial funding in Year 4 are referred to as Cohort 3; and those who received initial funding in Year 6 are referred to as Cohort 4.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Report Organization
Report Organization
The purpose of this report is threefold: 1) to provide details regarding only Cohort 3 participation for all three years of funding for it, 2) to provide details regarding only Cohort 4 participation for the only year of funding for it thus far, 3) to provide a cumulative summary regarding participation for all cohorts over all funding periods.
Data supporting the first purpose address the period of September 1, 2006, through August 31, 2009;data supporting the second purpose address the period of September 1, 2008, through August 31, 2009;and data supporting the third purpose address the period of February 2, 2004, through August 31, 2009. Participant data were collected through the Participant Background Survey, an instrument developed by the UMass Donahue Institute (UMDI) and administered by partnerships to each participant on the last day of each course. See Appendix A for the survey used during Year 6. The purpose of this survey is to gather data about participants’ professional backgrounds and qualifications. This information provides a picture of who the participants are, aids in determining whether the courses are reaching the teachers who most need professional development, and aids in tracking how teacher qualifications may change during the MMSP funding period. Data from the survey regarding teacher licensure, possession of and progress towards earning degrees, and status in terms of Massachusetts Tests for Educator Licensure (MTEL) exams allows determination of the number of teachers who meet criteria defining highly qualified status. Unless noted, data from the survey are reported in terms of unique individuals, regardless of the number of courses taken by each individual.
Data speaking to the strengthening of relationships between partnership members were collected through a section of the local evaluation reports that partnerships were required to submit to the ESE. In this section, partnerships were asked to describe the extent to which their courses had been integrated into activities of their higher education partners.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Evaluation Plan and Activities
Evaluation Plan and Activities
State-level Evaluation
Although not required by the U.S. Department of Education, the ESE contracted with UMDI to conduct a state-level evaluation of the MMSP. UMDI’s primary role as state-level evaluator is to coordinate program-wide collection of outcome data on behalf of the ESE. Data collection for the state-level evaluation is organized around a basic logic model for professional development initiatives shown below.
Local Evaluation and Related Technical Assistance
In addition to the state-level data collection, each partnership is required to conduct its own local evaluation. In an effort to support strong local evaluations, ESE required that partnerships sub-contract with UMDI to provide technical assistance on design and implementation of their local evaluations. The timeline listing the evaluation activities is found in Appendix B.
/ UMass Donahue InstituteResearch and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Cohort 3 Activity
Cohort 3 Activity: September 1, 2006 through August 31, 2009
Cohort 3, which began in the 2006-2007 funding period, consisted of nine partnerships. Table 1 shows the funding received by Cohort 3 partnerships for the period beginning in September 2006 and ending in August 2007, for the period beginning in September 2007 and ending in August 2008, for the period beginning in September 2008 and ending in August 2009, and for the cumulation of the three.
Table 1: Budgets: Cohort 3 PartnershipsPartnership / Sep06-Aug07 / Sep07-Aug08 / Sep08-Aug09 / TOTAL
EduTron Lowell (M/S) / $210,000 / $220,000 / $240,000 / $670,000
EduTron Fitchburg (M) / $102,000 / $110,000 / $120,000 / $332,000
Lesley University C3 (M) / $347,911 / $355,626 / $355,357 / $1,058,894
NorthShore (S) / $196,474 / $194,729 / $199,871 / $591,074
UMass Amherst C3 (M/S) / $107,424 / $216,281 / $169,064 / $492,769
SalemState College C3 (M) / $120,882 / $113,551 / $36,604 / $271,037
SE Cape (S) / $129,438 / $181,420 / $169,246 / $480,104
WPI – Science (S) / $99,586 / $70,734 / $94,852 / $265,172
Worcester PS (M) / $231,210 / $231,210
TOTAL / $1,544,925 / $1,462,341 / $1,384,994 / $4,392,260
State-level Participant Background Data
Cohort 3 consisted of nine partnerships, with four of the nine partnerships offering mathematics professional development, three offering science professional development, and two offering both mathematics and science professional development.
Across all three years of funding, 130 Cohort 3 courses were delivered. Of these 130 courses, 70 were mathematics courses, 58 were science courses, and two courses covered both math and science content. Of those 130courses, 64 (49%) were unique, and 66 (51%) were repeat offerings. This section of the report summarizes data collected from participants in these courses.
Across all three years of funding for Cohort 3, there were 1,076 participants, and 389 of them took two or more courses within Cohort 3. By the end of the 2008-2009 funding period, 1,076 unique participants completed the Participant Background Survey on one or more occasions. The term “unique participant” refers to each individual who participated in the program, regardless of how many courses he or she took. Data for items from the survey that help to convey participants’ professional backgrounds and motives for participation are discussed in the remainder of this section. All survey data for this group may be found in Appendix C. The responses to the survey questions are presented as frequencies and percentages. Not all percentages total 100% because many items allowed multiple responses and not all of the participants responded to all of the items.
Position of Participants
At the time of their last completed survey from a Cohort 3 MMSP course, 92% of course participants identified themselves as teachers. Of all respondents, 75% were regular education teachers; 11% were special education or special education inclusion teachers; 3% were ELL teachers; 2% were department heads or curriculum coordinators; 2% were principals, assistant principals, or headmasters; 1% were support specialists; 1% were paraprofessionals; 1% were long-term substitutes; <1% were superintendents or assistant superintendents; and 4% indicated that they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 2. Because respondents identified all teaching areas that applied to their positions at the time of the survey, some selected multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 38% of Cohort 3 participants were teaching mathematics (including elementary mathematics), 31% were teaching science, and 25% were teaching all subjects at the elementary level.
Table 2: Teaching Areas: Cohort 3 ParticipantsTeaching Areas
(Multiple responses permitted) / Sep06-Aug07 N=458 / Sep07-Aug08 N=477 / Sep08-Aug09
N=463 / TOTAL
N=1076
n / % / n / % / n / % / n / %
Mathematics / 153 / 33 / 161 / 34 / 125 / 27 / 326 / 30
Any science area / 151 / 33 / 213 / 45 / 143 / 31 / 337 / 31
General Science / 72 / 16 / 107 / 22 / 73 / 16 / 197 / 18
Biology / 24 / 5 / 25 / 5 / 36 / 8 / 70 / 7
Earth Science / 13 / 3 / 15 / 3 / 19 / 4 / 36 / 3
Chemistry / 18 / 4 / 27 / 6 / 34 / 7 / 64 / 6
Physics / 15 / 3 / 25 / 5 / 29 / 6 / 56 / 5
Technology/Engineering / 9 / 2 / 14 / 3 / 14 / 3 / 32 / 3
Computer Science / 5 / 1 / N/A / N/A / N/A / N/A / 5 / 1
Elementary(all subjects) / 148 / 32 / 117 / 25 / 150 / 32 / 265 / 25
Elementary Mathematics / 41 / 9 / 36 / 8 / 44 / 10 / 85 / 8
Other / 21 / 5 / 14 / 3 / 14 / 3 / 41 / 4
Not Currently Teaching / 19 / 4 / 39 / 8 / 38 / 8 / 84 / 8
Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 1,076 unique Cohort 3 participants was as follows: 17% were in their first to third year of teaching, 40% had between four and ten years experience in education, 28% had between 11 and 20 years of experience, 15% reported over 20 years of experience, and 1% did not report.
Teaching Levels of Participants
For reporting purposes, schools in the participating districts were organized into categories of elementary schools (grades K-5), K-8 schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last completed survey from an MMSP course, 34% of Cohort 3 participants were teaching in an elementary or K-8 school, 42% were teaching in a middle school, 15% were teaching in a high school, 1% were teaching at both the middle and high school levels, and less than 1% were teaching at either all levels or other levels such as pre-K or an adult level. Those remaining either were not currently teaching or the level at which they taught was unknown.
Types of Schools of Participants
As shown in Table 3, 97% of unique Cohort 3 participants worked in a public school setting, and 2% worked in a non-public school setting.
Table 3: Types of Schools of Unique Participants: Cohort 3 ParticipantsSchool Type / Sep06-Aug07 / Sep07-Aug08 / Sep08-Aug09 / TOTAL
n / % / n / % / n / % / n / %
Public Schools (includes public charter schools) / 447 / 98 / 462 / 97 / 452 / 98 / 1042 / 97
Non-publicSchool / 7 / 2 / 7 / 2 / 8 / 2 / 19 / 2
Other or No Response / 4 / <1 / 8 / 2 / 3 / <1 / 15 / 1
TOTAL / 458 / 100 / 477 / 100 / 463 / 100 / 1076 / 100
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for the high need designation.
The ESE expected that at least 50% of participants in each partnership would come from high need districts, and further, they set an informal goal that at least 75% of participants for each partnership would come from high need districts. Of all Cohort 3 participants, 72% came from high need districts.Table 4 shows that by the end of the 2008-2009 funding period, 74% of Cohort 3 participants from public schools had come from high need districts.
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 3 ParticipantsSchool Type / Sep06-Aug07 / Sep07-Aug08 / Sep08-Aug09 / TOTAL
n / % / n / % / n / % / n / %
High Need District / 343 / 77 / 335 / 73 / 339 / 75 / 772 / 74
Non-high Need District / 94 / 21 / 122 / 26 / 112 / 25 / 265 / 25
Unknown / 10 / 2 / 5 / 1 / 1 / <1 / 5 / 1
TOTAL / 447 / 100% / 462 / 100% / 452 / 100% / 1042 / 100%
Table 5 presents the number of participants from high need districts organized by each partnership. An examination of high need district participation in individual partnerships reveals that over the course of each partnership’s involvement in MMSP since the beginning of MMSP, six of the nine partnerships had at least 50% of their participants coming from high need districts across all years of their involvement and five of nine partnerships exceeded the informal goal of having at least 75% of the participants come from high need districts. If individual years of participation are examined, it is seen that six of the nine partnerships had at least 50% of their participants coming from high need districts for each and every year of funding, and four of the nine partnerships had at least 75% of the participants come from high need districts for each and every year of funding.