Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Reporting Period:
February 2, 2004 through August 31, 2006

Prepared for the Massachusetts Department of Education

March 2007

Annual Report of the MMSP State-level Evaluation / Contents

Contents

Executive Summary

Program Description

Evaluation Plan and Activities

State-level Evaluation

Local Evaluation and Related Technical Assistance

MMSP Participant Background Data

State-level Participant Background Data

Partnership-level Participant Background Data

Progress Toward Meeting MMSP Goals

Goal I.

Goal II.

Goal III.

Goal IV.

Goal V.

Discussion and Recommendations

Appendix A:Timeline for State-level Evaluation and TA Activities

Appendix B:Minimum Expectations for Evaluation

Appendix C: Participant Background Survey – Year 1

Appendix D: Participant Background Survey – Year 2

Appendix E: Participant Background Survey – Year 3

Appendix F: Results of the Participant Background Survey

Appendix G:High Need District Eligibility

Appendix H: Subject Matter Competency Demonstration Options

Appendix I: Highly Qualified Status of Unique Participants

Appendix J: Criteria that Account for Gain in Highly Qualified Status

/ UMass Donahue Institute
Research and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Tables

Tables Index

Table 1: Partnership Budgets by Program Year

Table 2: Teaching Areas

Table 3: Degrees Held and Pursued in Relation to Subjects Taught and Licensure

Table 4: Types of Schools of Unique Participants

Table 5: High Need Status of Unique Participants from Public Schools

Table 6: High Need Districts by Partnership

Table 7: Reasons for Participation in the Course – All Seats

Table 8: EduTron Participant Background Information

Table 9: Harvard Participant Background Information

Table 10: Lesley University Participant Background Information

Table 11: MCLA Science Participant Background Information

Table 12: Salem State College Participant Background Information

Table 13: Springfield/Holyoke Participant Background Information

Table 14: Wareham Participant Background Information

Table 15: WPI Participant Background Information

Table 16: MCLA Math Participant Background Information

Table 17: University of Massachusetts Amherst Participant Background Information

Table 18: Repeat Participants by Partnership

Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

Table 20: Highly Qualified Status of Unique Participants

Table 21: Number of Unique Participants Gaining Highly Qualified Status, by Partnership

Table 22: MTEL Tests Taken by MMSP Participants – Total to Date

Table 23: HOUSSE Plan Status of Unique Public School Teachers

Table 24a: MMSP Science and Technology Teaching Areas—Regular Education

Table 24b: MMSP Science and Technology Teaching Areas—Special Education

Table 25a: MMSP Mathematics Teacher Levels —Regular Education

Table 25b: MMSP Mathematics Teacher Levels —Special Education

Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains in Mean Scores

/ UMass Donahue Institute
Research and Evaluation Group / 1
Annual Report of the MMSP State-level Evaluation / Executive Summary

Executive Summary

The purpose of the Massachusetts Mathematics and Science Partnership (MMSP) Program is to provide high quality professional development for teachers in Massachusetts in the content areas of mathematics, science, and technology/engineering (MSTE). This multi-year project is funded through Title IIB of the No Child Left Behind (NCLB) Act. The intention of the MMSP funding is to increase the number of highly qualified teachers in the specified content areas in the Commonwealth, particularly in high need districts, with an ultimate outcome of increased student achievement. The funding is administered by the Massachusetts Department of Education (MADOE).

The Program began in February 2004, and has had three funding periods, defined as follows:

  • Year 1: February 2, 2004 through August 31, 2004
  • Year 2: September 1, 2004 through August 31, 2005.
  • Year 3: September 1, 2005 through August 31, 2006.

The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial funding in Year 2 are referred to as Cohort 2. Cohort 1 consisted of eight partnerships, with six of the eight partnerships offering mathematics professional development and two offering science professional development. Cohort 2 consisted of two partnerships, both offering mathematics professional development.

The University of Massachusetts Donahue Institute was contracted to coordinate state-level collection of outcome data and to provide evaluation-related technical assistance to the partnerships.

The MADOE established the following five goals for the partnerships to achieve through MMSP funding. Data supporting each goal were collected from February 2, 2004 through August 31, 2006, for both Cohort 1 and Cohort 2[1].

GoalI.Develop and implement an effective and sustained course of study for current teachers of grades 4-8 mathematics and/or grades 4-12 science and technology/engineering.

  • A total of ten partnerships were funded across the Commonwealth. Eight were organized around mathematical content, and two were organized around science content. Of the ten MMSP partnerships, all delivered courses.
  • In total, 85 MMSP courses were offered by the end of Year 3 of MMSP funding. Of these 85 courses, 76 were mathematics courses, and eight were science courses.
  • In total, 909 unique participants participated in MMSP courses by the end of Year 3. Of all 909 unique participants, 97% came from public schools (including 3% from public charter schools), 2% came from non-public schools, and less than 1% did not indicate their school type. Approximately 60% of participants from public schools were from high need districts.
  • There were 354 participants who took more than one MMSP course by the end of Year 3. Of the 354 participants who took more than one course, approximately 60% were from high need districts.

Goal II.Increase the number of teachers currently employed in the partnership school districts who are licensed in the areas they teach and/or have completed their HighObjectiveUniformState Standard of Evaluation (HOUSSE) plans.

Public school teachers must meet the federal definition of highly qualified to comply with the federal No Child Left Behind legislation. Goal II addresses the spirit of the federal No Child Left Behind legislation regarding teacher licensure, professional development, and competency in subject area taught.

  • Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
  • 67% of regular education middle and high school MMSP teachers who taught mathematics were licensed in mathematics.
  • 18% of special education middle and high school MMSP teachers who taught mathematics were licensed in mathematics.
  • 35% of regular education middle and high school MMSP teachers who taught a science or technology content area were licensed in the area in which they taught.
  • 3% of special education middle and high school teachers who taught a science or technology content area were licensed in the area in which they taught.
  • Highly Qualified Status
  • By the end of Year 3, of the participants who had entered MMSP as not highly qualified, 111 had attained highly qualified status. The breakdown of how highly qualified status was attained is as follows:

Sixteen passed the appropriate Massachusetts Tests for Educator Licensure (MTEL).

Five obtained a degree in content areas taught.

Three earned a teaching license.

Three obtained undergraduate degree equivalents in content areas taught.

Fourteen simultaneously met two or more criteria.

Seventy of the 332 who indicated they had HOUSSE plans completed a sufficient number of Professional Development Points (PDPs) on HOUSSE plans.

  • HOUSSE Plan Information[2]
  • Of 883 public school participants, 38% reported having a HOUSSE Plan, 32% reported not having a HOUSSE Plan, and 27% were unsure.
  • By the end of Year 3, of the 332 public school participants who indicated they had HOUSSE Plans, 253 (76%) reported having 48 or more PDP hours, 32 (10%) reported having fewer than 48 PDP hours, and 47 (14%) did not respond.

Goal III.Increase the number of highly qualified teachers in mathematics, science, and/or technology/engineering by integrating the courses of study into schools of education and/or arts and sciences at institutes of higher education.

For Year 3, partnerships were asked to describe the activities of their partnerships during the Year 3 funding period that spoke to the “institutionalization” of their courses, the extent to which their courses have been integrated into activities of their higher education partners. The extent and type of integration varied across partnerships. Following are summaries for partnerships that indicated that additional integration was occurring in Year 3:

  • In Year 3, all MMSP courses that EduTron created were offered through the Center for Professional Studies at Fitchburg State College. All courses were offered to in-service teachers for graduate credit. In addition, the partnership united Mathematics faculty and Education faculty at Fitchburg State College in a fundamental way: All parties are now working together to improve teacher preparation in mathematics through more stringent requirements in math courses and by improving mathematics offerings. The education department is in the process of hosting a math education summit to articulate the problems and present potential solutions that are arrived at through the joint efforts of math and education faculty.
  • In addition to continuing with the programs that were created in prior MMSP years, LesleyUniversity has begun developing an online mathematics program that will offer a master’s degree in elementary or middle school mathematics education. Through needs that have arisen through the classroom implementation component of their participation in MMSP, Lesley also has developed a relationship with Cisco Corporation who will be providing the online platform for the online mathematics program.
  • Massachusetts College of Liberal Arts (MCLA) Science now offers two new variable credit science for educators courses for in-service teachers.
  • Massachusetts College of Liberal Arts (MCLA) Math now offers one new variable credit math for educators courses for in-service teachers.
  • Salem State College continues to offer courses developed through MMSP as part of a master’s level teaching program in middle school mathematics. All courses developed by Salem State College through MMSP can be applied towards earning a degree through that program.
  • The three science courses offered by the Springfield partnership through the MMSP program in Year 3 were offered for graduate credit through University of Massachusetts (UMass) Continuing Education. A goal of the partnership is to have these courses institutionalized in the UMass School of Continuing Education through the School of Education.
  • All courses developed through the UMass Amherst partnership can be taken for graduate level credits that can be transferred to a variety of graduate level programs at UMass. In addition, the two courses are being reviewed by the School of Education Academic matters Committee for permanent course approval.
  • The one new course created by WPI in Year 3 was embedded into the master’s level graduate program in mathematics education offered by WPI’s Mathematics Department.

Goal IV. Increase the number of MSTE teachers currently employed in the partner school districts who participate in content-based professional development activities and substantially increase their content knowledge in order to be able to teach effectively the state learning standards.

  • Of the 85 courses offered across all partnerships by the end of Year 3, pre- to post-course test score gains were statistically significant in 79 (93%), indicating the courses had a positive impact on participants’ content knowledge.
  • Content assessments for three courses offered through one partnership were administered to both treatment (i.e., MMSP) and comparison groups. For each course, the treatment group post-course scores were statistically significantly higher than those of the comparison group, although in one case the treatment group pre-course scores also were statistically significantly higher than those of the comparison group.
  • 78% of the participants reported taking MMSP courses to increase knowledge in the content offered.

Goal V. Improve student academic achievement as measured by Massachusetts Comprehensive Assessment System (MCAS) mathematics and science and technology/engineering assessments and other assessments.

  • Partnerships reported student data directly to MADOE and to the United States Education Department under the federal reporting system.
  • The Donahue Institute state-level data collection effort for Title IIB did not include student outcome data; however, more than 89,000 students were taught annually by MMSP participants.

Data Quality

While in many instances for this project the data are sound, there are some areas for which this was not the case. The Donahue Institute became aware that problems existed with some of the data when 1) surveys received from participants taking multiple courses revealed inconsistencies in reporting across surveys for individual participants, especially for items used in determining highly qualified status, and 2) surveys indicated that many participants were confused about whether or not they had a HOUSSE plan.

The Participant Background Survey was modified after Year 1 to improve the likelihood of obtaining good quality data, but similar problems with the data arose again in Year 2. The survey was modified again in Year 3, but some problems still existed, and totals to date will be affected by problems from Year 1 and Year 2; therefore, information regarding the highly qualified status of teachers must be viewed with caution.

Reaching Targeted Participants

The partnerships exceeded the MADOE goal of having at least 50% of all participants come from high need districts. Half of the ten partnerships providing courses by the end of Year 3 met that goal as individual partnerships.

One obstacle that interfered with efforts to raise the high need district participation involved one partnership, in particular, that had committed to one cohort going through multiple courses over the life of the project. For this partnership, it was not possible to change their participants mid-program.

  • MADOE should continue to remind partnerships that enrollment must include a high percentage of participants from high need districts.
  • MADOE should encourage partnerships to recruit additional participants from their high need partners and/or identify additional high need partners, even if they meet the minimum participant rate of 50%.
  • MADOE has already begun revisiting the procedure by which districts are identified as high need districts. MADOE also should consider developing criteria to identify individual schools as high need.

Local Evaluation Designs

It was determined early in the initial meetings with partnerships that implementing a rigorous evaluation design – using randomized-controlled trials or strong comparison groups – would be difficult, if not impossible. The limitations that arose were limited resources, the lack of sufficient participant numbers to form reasonable treatment and control groups, and insufficient time to recruit participants into a control group before the start of the program. At the end of Year 3, only one partnership had implemented a weak quasi-experimental evaluation design. The remaining partnerships implemented basic evaluation models utilizing pre-post comparisons. To address the difficulties in creating rigorous evaluation designs, the Request for Proposals for MMSP programs for 2006-2007 included information on how partnerships might obtain additional support through MADOE and the Donahue Institute for this purpose.

  • MADOE and the Donahue Institute should continue to work closely with partnerships to provide technical support and resources to conduct rigorous evaluations in the continuation process.

Collecting Student Data

Perhaps the most pervasively problematic issue regarding evaluation of MMSP projects has been difficulty accessing student data. This has been both a logistical and legal challenge for evaluators. Logistically, many school districts do not routinely code their student data by teacher; thus, it is difficult to obtain data aggregated at the participant level. Partnerships have trouble enlisting the necessary cooperation from participating districts with few teachers enrolled in MMSP courses. Finally, Family Educational Rights and Privacy Act (FERPA) regulations present a real legal barrier in situations where the evaluation contract is not managed and paid for by the participating districts – the only relationship that would enable student-level data to be released to the evaluator. To date, it appears as if only two partnerships, EduTron and Lesley, have made a concerted effort to obtain student outcome data at the participant (as opposed to school) level.

Because student data was unattainable for most partnerships, they also encountered problems for meeting the federal reporting requirements regarding student data.

  • MADOE should give partnerships guidance on forging agreements with their partner districts to have access to student outcome data for participating teachers.

Integrating Courses into Higher Education Institutions

In the second year of the project, efforts were made to define integration and provide partnerships with structured guidance on reporting progress at integration. In the third year, structured guidance was again provided, and the data provided evidence that integration is occurring across almost all partnerships and that integration efforts will continue in the future for many of the partnerships, with at least one partnership (Lesley University) extending integration efforts beyond the scope of the MMSP grant. While the extent and types of integration that have occurred varied across partnerships, integration has happened primarily in the following ways that encourage sustainability beyond the duration of the program: embedding MMSP courses into pre-existing programs, integrating courses into pre-existing continuing education programs, and creating new degree programs.

  • MADOE should encourage partnerships to continue efforts to integrate MMSP courses into degree programs of partner higher education institutions.

Steering Committee

The Steering Committee has continued to provide guidance and feedback to the MADOE on issues regarding the MMSP partnerships, on broader issues of professional development and mathematics, science, technology, and engineering. The MADOE is fully aware how fortunate it is to have such an engaged and invested Steering Committee for this initiative. Steering Committee members have made observation visits to most of the partnerships and have offered programmatic suggestions and recommendations to the department.