An Evaluation of the Massachusetts Pilot Teachers’ SEI Endorsement Course

Erin Haynes, American Institutes for Research

Diana August, American Institutes for Research

Christine Paulsen, Concord Evaluation Group


An Evaluation of the Massachusetts Pilot Teachers’ SEI Endorsement Course

December 19,2012

Erin Haynes

Researcher, American Institutes for Research

Diane August

Managing Researcher, American Institutes for Research

Christine Paulsen

Principal Research Scientist, Concord Evaluation Group

1000 Thomas Jefferson Street

Washington, DC 20007

Phone: (202) 403-5000

Fax: (202) 403-5001

Copyright © 2012American Institutes for Research. All rights reserved.

3280_12/19

Contents

Executive Summary

Description of Evaluation

Research Questions

Recommendations

Introduction

Research Questions

Quality of Delivery

Fidelity of Implementation

Coherence

Teacher Outcomes

Research Context

Data Collection Methods

Activity 1: Course Observations

Activity 2: Instructor Interviews

Activity 3: Teacher Surveys

Findings and Recommendations

Quality of Delivery

Fidelity of Implementation

Coherence

Teacher Outcomes

Next Steps

References

Appendix A: Data Collection Instruments

Face-to-Face Session Observation Protocol

Online Session Observation Protocol

Instructor Interview Protocol

Participant Focus Group Protocol

Participant Survey

Appendix B: Participant Survey Results

Tables

Table 1. RETELL Pilot Sessions

Table 2. Research Questions Addressed by Each Data Collection Activity

Table 3. RETELL Pilot Session Observations

Table 4. Usefulness Ratings for Readings Assigned in the RETELL Course

Table 5. Topics for Which RETELL Participants Reported Greater Understanding

Table 6. Topics in Which RETELL Participants Need Additional Information

Figures

Figure 1. RETELL Course Participants’ Responses to Survey Questions About Course Readings

Figure 2. RETELL Course Participants’ Responses to Survey Questions About Instructor Quality

Figure 3. RETELL Course Participants’ Responses to Survey Questions About Online Sessions

Figure 4. RETELL Course Participants’ Responses to Survey Question About Course Pace

Figure 5. Average Time Spent on Activities During RETELL Course Face-to-Face Sessions

Figure 6. RETELL Course Participants’ Responses to Survey Questions About Time Allocation

Figure 7. RETELL Course Participants’ Response to Survey Question About Course Instructions

Figure 8. RETELL Course Participants’ Response to Survey Question About Time Allocated to Discussions of Classroom Strategy Application

Figure 9. RETELL Course Participants’ Response to Survey Question About Usefulness of Online Illustrations of Strategy Application

Executive Summary

Description of Evaluation

In order to improve instruction for English language learners(ELLs) in Massachusetts, the Massachusetts Department ofElementary and Secondary Education (ESE) has introduced the Rethinking Equity and Teaching of English Language Learners (RETELL) initiative. Under this initiative, all core academic teachers in the state will be required to complete comprehensive professional development (PD) in sheltered English instruction (SEI) methods by July 1, 2016.For simplicity in this report, this PD will be referred to as the RETELL course pilot, although the course will be called the Teacher SEI Endorsement Course in the future.

Implementation of the full program will begin across the state in February 2013. In order to ensure that this program is ready for complete rollout by February, a pilot of the PD was implemented during fall 2012, ending in November. The American Institutes for Research (AIR), in collaboration with Concord Evaluation Group (CEG), has completed an evaluation of the pilot to provide feedback on the quality of the RETELL PD and RETELL instructors’ fidelity to course materials in time to make any necessary changes to the program before the February rollout. A preliminary report of the findings of the evaluation was submitted to ESE in November. This document constitutes a full report of the RETELL pilot evaluation.

Research Questions

The RETELL pilot evaluation report responds to nine research questions, grouped into four critical categories for success of the PD program: quality of delivery, fidelity of implementation, coherence, and teacher outcomes. The research questions are as follows.

Quality of Delivery

  1. What is the overall quality of delivery of the online and face-to-face courses, in terms of content, process, and context, as well as pace, allocation of time, and clarity of PD instruction? How can the overall quality of delivery of the online and face-to-face courses be improved?
  2. In what ways could the support for RETELL course instructors be improved to enhance their instruction?
  3. What ongoing feedback loops can be established to ensure the quality of the RETELL instruction and course delivery?

Fidelity of Implementation

  1. What is the level of implementation of RETELL components during the online and face-to-face courses? How can the level of implementation of RETELL components during the online and face-to-face courses be improved?
  2. What ongoing feedback loops can be established to ensure the fidelity of the RETELL instruction and course delivery?

Coherence

  1. What is the level of coherence of the RETELL sessions?
  2. What is the level of coherence of the course with what teachers experience in their educational contexts?

Teacher Outcomes

  1. How could the RETELL course better contribute to key understandings of teachers of ELL students in regular classroom settings, in terms of both the content of the course and the course delivery?
  2. What additional information do teachers need to be successful in the RETELL course?

Recommendations

The following is the complete list of recommendations made in this report in each of the four identified critical areas: quality of delivery, fidelity of implementation, coherence, and teacher outcomes.

Quality of Delivery

Quality of Content

  • Goals, objectives, and expected outcomes appear in the course syllabi and presentation slides. Participants seem capable of assimilating the information from course materials. Therefore, no change is recommended.
  • No changes are recommended to the topics covered in the course syllabi. If changes are made to the syllabi for other purposes (e.g., to ensure that participants have more opportunities to engage with specific strategies), make sure that key areas of ELL instruction (see description and sources in full report) are retained.
  • Teacher buy-in is essential in order to motivate teachers to adopt practice, especially when the PD does not already align with the beliefs of the teacher (Kubitskey & Fishman, 2005). Because some recipients of the RETELL PD may not be convinced that they can successfully impart content to students who are not proficient in English, consider adding a case study or video to the first session of the course that illustrates how sheltering instruction can help ELLs access content at the same time as they acquire English language skills.
  • To reduce printing burdens for teachers, provide a CD or printed copies of the required readings.
  • Reduce the reading load by carefully selecting readings that are the most relevant to the successful implementation of SEI and that are clearly written. Consider readings that provide practical advice about helping ELLs meet high content standards.

Quality of Process

  • The pilot instructors expressed knowledge of the course participants’ backgrounds. This knowledge is important to the quality of process, and instructors of the main course should learn about their participants’ backgrounds and contexts. Consider including some short activities in the course that explore what participants already know about various topics and what they would like to learn. Provide sufficient flexibility for instructors to be responsive to participants’ needs.
  • Identify instructional strategies that are aligned with each course topic. Have PD instructors model these strategies using elements of model lessons and give participants an opportunity to apply what they have learned in a new context. Consider providing participants with a rubric to record each strategy they use, with information about the classroom context, how the strategy worked, and recommendations for adjustments.
  • Ask participants to briefly summarize what they learned at the end of each session, either as a whole class, in small groups or partners, or as a warm-up during a subsequent online session. As part of their summary, ask them to link the course materials to specific experiences or examples from their own educational contexts, or describe methods they have used related to the session.
  • Pilot instructors appeared to be skilled at leading collaborative discussions and engaging participants. Ensure that instructors in future RETELL courses are prepared to use good pedagogical practice to facilitate collaboration and encourage participation.
  • Consider developing professional learning communities (PLCs) as part of the RETELL course so that collaboration among participants can extend beyond the duration of the course. A review of relevant research on PLCs by Vescio et al. (2008) finds that well-developed PLCs have positive effects for both teacher practice and student outcomes.
  • Consider methods for increasing discussions and interactions online. Some options include the following:
  • Establish weekly “office hours” during which the instructor is online and available for chatting (consider making attendance at some office hours a course requirement).
  • Create a blog or wiki in which participants can report on the implementation of strategies or share ideas for teaching that came to them as a consequence of a reading or session.
  • Group participants and provide tasks that require them to check in with each other regularly, either in person or via the online platform.
  • Set up a collaborative project online to which all participants are required to contribute. Ideally, such a project would be best facilitated by the online platform (such as posting videos of instruction and eliciting constructive feedback or conducting an analysis of student work using graphic tools; see Little et al., 2003).
  • Encourage instructors to actively monitor online activity and send reminders and prompts to participants as needed.

Quality of Context

  • It will be essential to select highly qualified instructors for a course that is not heavily scripted. Unless activities are added and scripted, only RETELL instructors who are prepared to develop and incorporate their own activities should be selected.
  • Consider developing and providing answer guides for questions posed in the course materials.

Pace of Course

  • Carefully monitor the semester-long course to make sure that the pace is not too fast. For example, the schedule should allow reasonable time between sessions for participants to complete readings and assignments, and to test the methods and instructional strategies they are learning.

Allocation of Time in Course

  • Because participants found course time allocations appropriate for lectures and in-class interactions with colleagues (including small-group work and whole-class discussion), maintain these activities at their current levels.
  • In order to address instructor concerns and participant feelings that not enough time was devoted to strategies, consider replacing some of the small-group discussions with small-group strategy practice; whole-class discussions also could be used to discuss how theory is realized in practice and how to apply instructional strategies successfully.
  • Replace significant out-of-class assignments with activities that require collaboration with colleagues, particularly on the practice of strategies. Be sure to incorporate enough time between assignments to allow participants to engage in these activities meaningfully.
  • To address time allocation during the self-paced, self-directed online sessions, consider creating an activity that helps participants understand how they learn best. Provide tips for time management as part of the introduction to the online sessions.

Clarity of Instruction

  • Consider including some short activities in both the face-to-face and the online portions of the course that explore participants’ conceptions and misconceptions to ensure that they are addressed.
  • For the online sessions, provide a single list of required readings, materials, and activities for each session with hyperlinks to each.
  • Include summaries of readings and information about possible questions or concerns in online materials.
  • Provide a list of assignments and readings ahead of the course so that participants have additional time to complete the work.

Support for Course Instructors

  • Make sure that course classrooms are easily accessible, can be accessed ahead of each session for setup, and have Internet access. If possible, designate a person at each site to ensure that these logistics are taken care of.
  • Provide participant contact information to instructors as early as possible (ideally, at least two weeks before the first session).
  • Establish a clear policy about material distribution and provide it to all instructors. Include information about which materials and services ESE provides and which it does not.

Establishment of Feedback Loops

  • Provide information to instructors about which aspects of the RETELL course will need to be consistent across the state, such as participant grading or evaluation information.
  • Consider providing a training session for RETELL course instructors during which they can practice elements of the PD with each other before delivering it to course participants.
  • Establish a consistent feedback system for all RETELL PD courses. The feedback system might consist of an observation by an ESE official, formal participant surveys (e.g., short surveys delivered via Survey Monkey after each class), or an outside evaluation of teacher products (e.g., lesson plans) or outcomes.

Fidelity of Implementation

  • Strategically group participants with similar backgrounds and educational contexts into the same courses so that instructors can focus on their needs efficiently, increasing the likelihood of covering all course content. As an added advantage, grouping participants from similar educational contexts can facilitate the formation of PLCs (see, e.g., Newmann & Associates, 1996).
  • Check online sessions to ensure that they cover all topics and assignments in the syllabus, especially if there have been changes to the sessions.
  • Provide instructors with information about how long each activity should take.
  • Facilitate regular meetings among instructors (by region and/or participant type—e.g., secondary STEM) throughout the course semester to discuss RETELL delivery and fidelity to the syllabus. These meetings would also be a good opportunity for ESE to collect feedback from instructors about recommended changes.
  • Make necessary changes to the course between, not during, rollout semesters.

Coherence

Level of Coherence of the RETELL Sessions

  • Revise course materials to provide explicit connections between topics covered in each session, including revising the online materials to include explicit connections to previous and upcoming sessions.
  • Explain to instructors the importance of communicating expectations for the online course during the face-to-face sessions, even if there is only a little time to do so.
  • Consider implementing the following instructor-recommended changes:
  • Incorporate session 4 into session 5.
  • Consolidate information about academic language into a single session.
  • Incorporate information about the Common Core State Standards throughout the sessions rather than introducing it in session 15.
  • The topic of assessment should come at the end of the course, after reading and writing have been covered.
  • Provide more grade-level differentiation in Module C [within the secondary courses].

Level of Coherence With What Teachers Experience in Their Educational Contexts

  • Connections to participants’ educational contexts are crucial. Ensure that RETELL course instructors are as prepared as the pilot instructors to connect material to real-world contexts, use stories to link theory to practice, and lead participants in discussions of how strategies would work in their educational contexts.
  • Consider incorporating the practice of instructional strategies into small-group work and whole-class discussions in order to increase its impact.
  • Update the online materials to make better connections to participants’ educational contexts. In each session, include
  • Stories or videos that model practical strategies
  • Discussions of possible challenges that teachers might face in applying strategies
  • Prompts that encourage participants to post and discuss examples of how strategies would work in their educational contexts

Teacher Outcomes

  • Conduct a brief survey at the outset of each course to determine areas in which participants are already well versed and areas where they need additional instruction.
  • Consider providing (additional) instruction in the following areas, as determined by participant needs:
  • Creating lessons and practical strategies for the classroom
  • Effective practice with ELLs who do not speak Spanish
  • Standardized assessments and ELLs
  • ELLs with special needs
  • Grade-specific strategies
  • Supporting language development versus helping students access content
  • Students with interrupted formal education (SIFE)
  • Using data
  • Fishman et al. (2003) propose an iterative PD research design model, in which feedback about teacher outcomes is incorporated into future PD sessions; a similar model has been proposed by Kubitskey & Fishman (2007). The current evaluation gathers self-reported data from teachers, but it also is important to measure teachers’ skills and knowledge. The following formative and summative evaluation activities are recommended both to provide teachers with information about their progress in the course and to improve future iterations of the course:
  • Develop and incorporate protocols for assessing participants’ plans for and reports of how they implemented course strategies.
  • Develop and conduct pre- and post-course assessments of participants’ knowledge about ELLs and ELL instruction.
  • Include a classroom observation component in the course. Teachers might observe each other and provide peer-to-peer feedback; a school or district ESL coordinator or coach might observe participants and provide feedback as part of the RETELL course; or RETELL course instructors might observe each participant teaching. Observations are recommended for teachers in high- and medium-incidence schools; they are not recommended if participants do not have ELLs in their classrooms at the time of the RETELL PD.
  • Make sure that instructors can provide timely feedback to participants using the Moodle online platform. For instructors who are unfamiliar with Moodle, offer training on how to use it effectively.

[This page intentionally left blank.]