EDPACE Year Two Study

Findings from the Implementation and Impact Studies

Submitted by Rockman et al

Overview

In Year 2 of the West Virginia EDPACE Project, the evaluation team continued its research on the implementation and impact of the VS (VS) program. The impact or summative research was designed to measure students’ Spanish proficiency and performance on non-Spanish assessments; the implementation study was designed to provide contextual information for that research, and to get a comprehensive picture of implementation statewide and document similarities and differences across sites. During the first year of the research studies, we developed survey instruments to collect data about implementation, made exploratory site visits to a sample of virtual and face-to-face sites, and developed a Spanish outcome measure to use in evaluating the Spanish language proficiency of students in both virtual and face-to-face sites. Our analyses of the data showed that there were few differences between virtual and face-to-face sites in the development of Spanish language proficiency, but there was a range of performance among the virtual sites. This range led us to design a second-year study that would allow us to look more closely at implementation and to take a second look at students’ performance on the Spanish proficiency outcome measure.

Methods

Research Design

The large quantity of qualitative data collected during site visits in Year 1 as well as the results of surveys were synthesized for the identification of emerging issues, patterns, factors, and themes that would constitute ones to probe more deeply with a set of targeted sites during the second year. In the process, we developed a logic model and mapped our research hypotheses and questions to clarify the data that would need to be collected in Years 2 and 3. (See Figures 1 and 2.)

In selecting sites for the in-depth, “targeted” study, we reviewed the data collected in Year 1 and created matrices of factors related to implementation. (See Figure 3 for template used to consider factors related to implementation and achievement.) We wanted a representative sample, based on factors such as geographical location, years in the project, and student achievement (as measured by the 1B Spanish written and oral assessments). We wanted to include sites with both 1A and 1B classes, as well as a few new sites that, though they had only 1A in the 2004-2005 school year, would add 1B the following year. We factored other administrative information into the selections, such as number of students in each class, and division of classes between both lead teachers and several adjunct teachers.

ED PACE Report, Page 1 of 1

Figure 1 VS Program Logic Model

I. Program II. VS III. VS IV. Intermediate V. Long term

Supports Program Design Implementation Outcomes Outcomes

o  Building State Capacity for Scientifically-based Research on Technology Interventions in Education

Figure 2 Alignment of EDPACE Research Questions, Constructs, and Data Sources

/

Constructs

/

Data Sources

All VS Sites / Targeted Sites Only / 1B Only
Program Design /

Research Questions

/ Facilitator Survey / Adjunct Survey / Student Survey / Observation Protocol / Parent Survey / Principal Interview / Screener / Spanish Assessment
Team Background and Dynamic / To what degree do differences in the background and experience facilitators and adjunct teachers bring to the VS program influence how they implement the program?
§  How many years have facilitators and adjunct teachers worked with the VS program?
§  What subjects/content areas do facilitators currently teach?
§  How do facilitators rate their technology skills?
§  How do facilitators rate their current Spanish proficiency, in comparison to their initial proficiency?
§  How does facilitator or adjunct background or experience affect how the program in implemented (e.g., the use of Spanish used in teaching, language use activities, / Instructional Team
(II.A.) / ü / ü
How do members of the instructional teamwork together to provide instructional support to students in the VS program?
§  How do members of the instructional team view their roles and the team dynamic?
§  To what degree do team members have a shared understanding of their roles and responsibilities and the goals of the program?
§  How do team members provide feedback to each other and to students?
§  Do team members believe that the team approach works?
§  What adjustment would they make in the roles and responsibilities? / ü / ü
Technology and Curriculum / How do the elements of the blended model work in different sites?
§  What percentage of activities are technology-based and non-technology-based in the VS classroom?
§  Do facilitators adhere to the calendar and lesson plans?
§  Does the technology function as expected?
§  What are the challenges of technology administration/maintenance? /

Blended Delivery Model

(II.B) / ü / ü / ü
Enrollment and Reception / How does the administration of the VS program function across sites?
§  What enrollment criteria do individual sites use for VS 1A and 1B?
§  How have principals/parents/students/and staff received or reacted to the program?
§  What are the logistical/administrative challenges at the site level? / Administration and Support
(II.C.) / ü / ü / ü
/ /

Research Questions

/

Constructs

/ Facilitator Survey / Adjunct Survey / Student Survey / Observation Protocol / Parent Survey / Principal Interview / Screener / Spanish Assessment
Student Background / What is the VS classroom composition, and does it vary from site to site?
§  What are the demographics of the students? (Gender, ethnicity, etc.)
§  What previous language learning and experiences do students have?
§  What home students do students have for language learning?
§  What is students’ prior achievement, and how does that affect their performance in VS? / ü / ü
Implementation / Classroom Organization and Facilitator Behaviors / To what extent do differences in classroom organization and time on different activities influence student engagement and subsequent performance in the VS classroom?
§  How much time do students spend working in pairs, groups, or alone?
§  How much time do they spend on computer-based activities?
§  How much time do they spend using other technologies?
§  How much time do they spend on non-technology activities?
§  How and to what extent do facilitators and students use other classroom resources?
§  Does the amount of time spent in various activities vary from site to site or 1A to 1B?
§  Are differences in time and classroom organization related to differences in student Spanish proficiency? / Classroom Processes and Activities
(III.A.) / ü / ü / ü / ü / ü
How does the facilitator manage classroom activities?
§  How do facilitators keep students on task?
§  What proportion of their time is spent on management activities? / ü / ü
Instructional Strategies / What impact do the team members’ behaviors and instructional strategies have on learning, proficiency, and engagement?
§  How do facilitators guide students’ learning of language content and skills?
§  To what degree do facilitators model co-learner behavior?
§  What roles do other team-members play in the on-site portion of the VS classroom?
§  What activities do lead teachers facilitate during telephone-mediated interaction with students?
§  What role do facilitators play during telephone time?
§  How and how often do adjunct teachers communicate with or visit classrooms?
§  To what degree do the facilitator, adjunct, and lead teacher collaborate to foster learning? / ü / ü / ü
Use of Resources / To what extent do students use technology resources and engage in non-technology activities?
§  How do students use the VS platform?
§  How and how often do students use Wimba voice tools?
§  How frequently do students refer to dictionaries, flash cards and other print materials?
§  Do facilitators make add-on materials available? / ü / ü / ü

Research Questions

/

Constructs

/ Facilitator Survey / Adjunct Survey / Student Survey / Observation Protocol / Parent Survey / Principal Interview / Screener / Spanish Assessment
Student Behaviors / In what ways are students involved in classroom activities and processes?
§  How engaged are students in classroom activities? What is the level of engagement in technology-based activities as opposed to non-technology activities?
§  How much time do students spend monitoring or directing their learning?
§  What do students do during telephone time with the lead teacher?
§  How often do students assist their peers with content or technology?
§  How often do students ask questions about content or language?
§  To what degree is students’ level of participation in learning activities related to Spanish proficiency and performance? / ü / ü / ü / ü
Language Instruction / What is the relationship between language use by the instructional team, and the blended approach, and students’ engagement and performance?
§  How often and for what reasons do facilitators use Spanish?
§  How often do lead teachers use Spanish during telephone time?
§  How and for what reasons do adjunct teachers use Spanish in interactions with students?
§  How often and in what ways do facilitators address aspects of cultural connections and language comparisons?
§  How do facilitators use classroom technologies and resources? / Language Instruction and Learning
(III.B.) / ü / ü / ü
Students’ Use of Spanish / What is the relationship between students’ language use and students’ engagement and performance?
§  How much time do students spend on language learning activities (reading, writing, listening, speaking)?
§  How often do students use Spanish in the real and virtual classrooms?
§  What language activities do students like? Which ones do they find challenging?
§  How confident are students in their Spanish skills? / ü / ü / ü / ü
Feedback and Support / How does communication in the VS program, especially online communication, and provide feedback to students and support learning?
§  What is the level and nature of teacher-teacher interaction?
§  What is the level of teacher-student interaction?
§  What opportunities do students have to interact with one another and their virtual amigos? / Collaboration and Community
(III.C.) / ü / ü / ü / ü / ü
Sense of Community / How does a sense of community develop in the VS classroom, and how does it contribute to engagement and performance?
§  What activities help develop a sense of community?
§  Do levels of communication between the instructional team, and the team and students, vary from site to site?
§  How much time do students spend interacting with one or more students?
§  How do facilitators or lead teachers promote classroom interaction? / ü / ü / ü / ü

Research Questions

/

Constructs

/ Facilitator Survey / Adjunct Survey / Student Survey / Observation Protocol / Parent Survey / Principal Interview / Screener / Spanish Assessment
Use of Technology / Is the level of technology use in VS related to student engagement?
§  What types of technology are used in various classroom activities?
§  How confident are students’ and facilitators’ in their technology skills, and does skill level affect implementation and learning?
§  How much class time is spent on technology-based activities?
§  What types of language skills do students gain through technology?
§  How do students use technology at home? / Use of Technology
(III.D.) / ü / ü / ü / ü
Intermediate Outcomes / Motivation / Which VS activities and features appear to motivate students to learn a foreign language?
§  What activities do students say they find most enjoyable and most challenging?
§  Which activities do facilitators say engage students or hold their attention? (technology, non-technology, hands-on, assessment)?
§  Does students’ engagement vary from 1A to 1B, or from site to site? / Student Engagement
(IV. A.) / ü / ü / ü / ü / ü
Participation / What is the relationship between students’ participation in communication and learning activities and performance?
§  Does level of enjoyment or challenge affect students’ participation in various activities?
§  How frequently do students take part in or initiate online communication?
§  What are some of the reasons they contact the instructional team? / ü / ü / ü / ü
Work completion and Communication / To what extent do students take responsibility for their own learning?
§  How much time do students spend monitoring their own learning?
§  How do adjunct teachers describe students’ ability to complete assigned activities?
§  How do adjunct teachers describe their to online communication from the instructional team?
§  Do parents believe that their children are committed to doing their best? / Self-direction and Responsibility
(IV. B.) / ü / ü / ü
Importance and Interest / What is the impact of the VS program on how much students value learning a foreign language?
§  Does the VS program encourage students to value learning and speaking a foreign language more?
§  Does the program encourage an interest in continuing to study foreign languages? / Valuing Foreign Language
(IV. C.) / ü / ü / ü
Frequency, Range of Use / To what extent do students use Spanish beyond the classroom?
§  How frequently do students use Spanish?
§  In what contexts (restaurants, with friends) do students use Spanish? / Spanish Use beyond the Classroom
(IV. D.) / ü / ü / ü

Research Questions

/

Constructs

/ Facilitator Survey / Adjunct Survey / Student Survey / Observation Protocol / Parent Survey / Principal Interview / Screener / Sp. Assmt./ WESTEST
Students / How satisfied are students with the VS class?
§  How satisfied are students?
§  What do students say about taking another Virtual class?
§  According to students, would they suggest the class to a friend? / Program Satisfaction
(IV. E.) / ü / ü / ü
Parents / How satisfied are parents with the VS class?
§  Do parents indicate that they’re happy with the key elements of the program—communication with teachers, technology, and level of challenge? / ü / ü / ü
Long-term Outcomes / Language Skills / What is the impact of the VS program on students’ Spanish performance?
§  What is the impact on reading, writing, speaking, and listening proficiency?
§  How does students’ performance compare to that of students in face-to-face classes? / Spanish Learning and Proficiency
(V. A.) / ü
Perceptions of Learning / How do students self-assess their own language skills? / ü
Standardized Assessments / Does participation in the 1A, 1B VS program have an impact on students’ general achievement?
§  Does participation in VS have an impact on students’ performance on standardized assessments in language arts and math?
§  How do VS students perform compared to students who have had no foreign language instruction? / Improved General Achievement
(V. B.) / ü
Enrollment and Course selection / What is the impact of the VS program on students’ participation in high school Spanish II or other course selections?
§  Do students participating in the middle school VS program enroll in high school Spanish II?
§  Does participation in middle school VS influence other high school course selections? / High School (Spanish) Participation
(V. C.)

ED PACE Report, Page 1 of 1