Preproposal: Integration Of

Abstract Title Page

Title:

Experimental Evaluation of a Scale-up Model for Teaching Mathematics with Trajectories and Technologies

Authors:

Julie Sarama, Douglas H. Clements, Mary Elaine Spitler, and Alissa Lange,

University of Buffalo, State University of New York

Experimental evaluation of preschool mathematics,

Abstract Body

Background/context:

Although the successes of some research-based educational practices have been documented, equally recognized is the “deep, systemic incapacity of U.S. schools, and the practitioners who work in them, to develop, incorporate, and extend new ideas about teaching and learning in anything but a small fraction of schools and classrooms” {Elmore, 1996 #1859, p. 1; see also \Berends, 2001 #1856; Cuban, 2001 #2085; Tyack, 1992 #1548}. There may be no more challenging educational and theoretical issue than scaling up educational programs across a large number of diverse populations and contexts in the early childhood system in the U.S., avoiding the dilution and pollution that usually plagues such efforts to achieve broad success. We created a research-based model to meet this challenge in the area of mathematics, with the intent the model generalize to other subject matter areas and other age groups.The field needs transferable, practical examples of scale up {McDonald, 2006 #2808}; empirical evidence of the effectiveness of these examples; and focused research on critical variables—all leading to refined, generalizable theories and models of scale up. Our research plan describes a project designed to meet those needs.

TRIAD’s theoretical framework (Sarama, Clements, Starkey, Klein, & Wakeley, 2008) is an elaboration of the Network of Influences model (Sarama, Clements, & Henry, 1998), illustrated in Figure 1 (please insert figure 1 here). It is consistent with, but extends in levels of detail, such theories as diffusion theory and the overlapping spheres of influence (Rogers, 2003; Showers, Joyce, & Bennett, 1987). Thus, scale up can be seen as the effort to maintain the integrity of the vision and practices of an innovation through increasingly numerous and complex socially-mediated filters, through phases of introduction, initial adoption, implementation, and institutionalization.

Purpose:

There are three primary research questions:

1. Can the intervention be implemented with high fidelity and have substantial positive effects on teachers’ practice beliefs and knowledge? Previous research indicated that most teachers implemented the curriculum with high fidelity and made significant gains in knowledge {Clements, 2008 #2785; Sarama, 2008 #3324}. We evaluated whether all aspects of the intervention can be replicated in greater numbers of classrooms, in distant, diverse sites. That is, do teachers participate in the complete professional development program, receive the prescribed support, implement the curriculum with high fidelity, and provide a richer classroom environment for mathematics? Do their knowledge and beliefs about early childhood mathematics develop as a result of the intervention? Do parents receive and implement mathematics activities?

2. What are the effects of the scale up intervention, as implemented under diverse conditions, on achievement and the achievement gap? In our current IERI project, children in the experimental group outperformed children in the control group in math achievement at the end of pre-K. We determined whether these findings are replicated with the same effect size under full scale up. Do experimental children from low-income homes and ethnic and linguistic minority groups narrow the achievement gap that separates them from children from middle- and high-income and non-minority homes?

3. How well may the model apply to varied settings for scale up efforts? Under what conditions can our findings on the impact of the intervention be generalized to the larger population and distinct contexts?

Setting:

The study took place in pre-K classrooms in two urban school districts, the Buffalo Public School system in Buffalo, NY and the Boston Public School system in Boston, MA.

Participants:

In the Buffalo Public Schools, all schools whose pre-K teachers had not previously been involved in Building Blocks {e.g., \Clements, 2008 #2785; Clements, 2007 #2091; Sarama, 2002 #1922; Sarama, 2004 #1715} or TRIAD {Sarama, 2008 #3324} research or development projects were included. Extending the evaluation to distal sites— essential for generalizing to the target population of all U.S. pre-Ks—all Boston, MA schools that were not adopting a new pre-K curriculum that year and whose principals agreed to participate were included. There was only 5% attrition of children included in the study (from control, 12 from Buffalo and 6 from Boston from TRIAD, 37 in Buffalo and 15 from Boston schools, for a total of 70), most moved out of state, but some were ill for the entire posttest period, leaving a total of 1305 children with complete data on both pretest and posttest).Analyses revealed no significant difference between those who left and remained in mean pretest achievement, nor any significant interaction between attrition and treatment group; finally, the small effect size, ES < .01, indicates that any effect on the findings was negligible.

Intervention:

The specific goal of our implementation of the TRIAD (Technology-enhanced, Research-based, Instruction, Assessment, and professional Development) model is to increase math achievement in young children, especially those at risk, by means of a high-quality implementation of the Building Blocks math curriculum. This includes all aspects of the curriculum—mathematical content, pedagogy, teacher’s guide, technology, and assessments—based on a common core of learning trajectories. Building Blocks, the I in the TRIAD acronym, was based on a comprehensive Curriculum Research Framework (Clements, 2007) and its efficacy validated by two Cluster Randomized Trial (CRT) evaluations, yielding effect sizes ranging from .5 to over 2 (Clements & Sarama, 2007, 2008). The Assessment component of TRIAD includes both formative assessment performed by the teachers training to use learning trajectories for this purpose, supplemented by the Building Blocks Software management system. TRIAD’s professional Development includes multiple forms of training (15 full days over two years, only the second of which included child assessments) and support (coaching and mentoring). Each of these uses the software application, Building Blocks Learning Trajectories (BBLT), which presents and connects all components of the innovation. BBLT provides scalable access to the learning trajectories via descriptions, videos, and commentaries. The two main aspects of each learning trajectories—the developmental progressions of children’s thinking and connected instruction—are linked to the other (see Fig. 2—please insert Figure 2 about here).

Research Design:

In a CRT design, schools within each district were ordered on the basis of their average scores on state-based math achievement tests and then publicly assigned to one of three treatment groups using a randomized block design (using a table of random numbers, with blind pointing to establish the starting number). We attempted to implement components of TRIAD implemented in ways that would be available under normal condition. The first year was a pilot/training year, as our current work confirms that of others indicating that teachers take at least two years before they begin implementing the curriculum conceptually and completely. Assessors were trained and validated during this time.

Data Collection and Analysis:

All assessments were completed the second year, including two measures of teachers’ classroom practices (e.g., implementation fidelity), knowledge, and beliefs—Fidelity of Implementation, Classroom Observation of Early Mathematics Environment and Teaching (COEMET) and two groups of measures of child outcomes, math (Research-based Elementary Math Assessment, REMA), and language and literacy (Renfrew Bus Story; PALS-PreK; MCLASS: CIRCLE).

To answer question 1, Can the intervention be implemented with high fidelity and have substantial positive effects on teachers’ practice, beliefs and knowledge?, factorial repeated measures analyses were conducted on the Fidelity (intervention groups) and COEMET (both groups) T-scores. (A Teacher Questionnaire was also employed to collect data on demographics and teacher beliefs.)

The other two questions were answered with hierarchical linear modeling {HLM, \Raudenbush, 2000 #2123; Raudenbush, 2003 #2119}. All level-2 predictors were centered around their group means. All interactions were computed on mean-centered transformations of the variables involved. Effect sizes were computed for significant main effects by dividing the regression coefficient by the pooled posttest standard deviation.

Findings/results:

HLM analyses revealed that the two groups differed significantly in math achievement and that both improved significantly and substantially (more than 1 SD). The TRIAD group outperformed the control group (p < .0001), with an effect size of .69. There were no significant main effects or interactions for SES or LEP (percentage of students with Limited English Proficiency). At the child level, there were no significant interactions for sex or disability status and only one interaction for ethnic group—African American children in the control group learned significantly less than other children in the control group, but African American children in the TRIAD group scored significantly learned significantly more than other children in the TRIAD group.

Fidelity scores were relatively high (on Likert scale items, averaging “agree”). COEMET scores were significantly higher for the TRIAD group at both time periods, Fall and Spring (p < .0001, ES = 1.13). COEMET scores were a significant mediator, although use of the Building Blocks curriculum alone accounted for more variance.

There were no differences between the groups on letter recognition, or on three measures from the oral language (Bus Story) measure: sentence length, listening, and story duration. The TRIAD group outperformed the control group on three language measures: information (ES = .33), complexity (ES = .16), and independence (ES = .36).

Conclusions:

High levels of fidelity of implementation resulted in consistently higher scores in the intervention, compared to control, classes on the COEMET and statistically significant and substantially greater gains in children’s math achievement in the intervention, compared to the control, children. The greatest treatment effects on the observation instrument were on several classroom culture items (responsiveness to children, use of teachable moments, and especially use of computer to teach math), and items regarding specific math activities (the number of activities, eliciting and extending children’s mathematical thinking, and especially observing and listening to children and using formative assessment).

TRIAD had a positive effect on math achievement. Probably due to both districts’ new emphasis on PreK math, the control group gained a surprising standard deviation (especially compared to national PreK math gains). Thus, the effect size of .69 for TRIAD is noteworthy. Teachers teaching more and better math accounted for some of the children's gains; just using the curriculum accounted for the rest.

Extensive time spent on mathematics in prekindergarten does not appear to hinder early literacy and language performance. Children exposed to the Building Blocks early math curriculum did not differ significantly from children who participated in the regular district math curriculum on letter recognition or on a number of language measures. Children who received the Building Blocks curriculum outperformed the control group on two language measures, information and independence. The Building Blocks group had an advantage over the control group in recall of key content from the narrative, a skill that has been linked to academic success at age 15. The higher independence scores for children in the Building Blocks group indicates that may be more confident than the control group in verbalizing their thoughts. Aspects of the curriculum, such as the focus on verbal explanations for solutions to math problems, may account for these differences. Results on these two measures suggest that children who learn math through Building Blocks may also develop important competencies that can transfer to other academic areas, such as oral language.

The theory of scale up employed held up under a CRT experiment. Centering on a common core of learning trajectories and the combination of workshops, coaching, mentoring, and Internet tools focused on a common curriculum, yielded a successful scale up with effects not dissimilar to ideal conditions.

There are four basic recommendations. (1) Curriculum and policy should ensure that children, especially those living in poverty, should be provided with research-based, focused early mathematical interventions which can increase their knowledge of multiple mathematical concepts and skills (including, but also going beyond number) without harming—and actually increasing—their language and literacy skills. (2) Substantial professional development may be necessary—15 full days were barely adequate to the task in this study, consistent with previous research (cf. an average of 53 hours of teaching training yielding an average effect size of .53 on student math achievement in Yoon, Duncan, Lee, Scarloss, & Shapley, 2007), suggesting a minimal duration for effective professional development. (3) The Curriculum Research Framework (Clements, 2007) upon which the curriculum was based has been repeatedly empirically supported and may serve as a useful guide to policy makers, curriculum and software developers, and administrators. (4) The learning trajectories at the core of the curriculum and TRIAD model may constitute a useful construct in future research, curriculum development, and professional development efforts.

Appendixes

Appendix A. References

Clements, D. H. (2007). Curriculum research: Toward a framework for ‘research-based curricula’. Journal for Research in Mathematics Education, 38, 35–70.

Clements, D. H., & Sarama, J. (2007). Effects of a preschool mathematics curriculum: Summative research on the Building Blocks project. Journal for Research in Mathematics Education, 38, 136-163.

Clements, D. H., & Sarama, J. (2008). Experimental evaluation of the effects of a research-based preschool mathematics curriculum. American Educational Research Journal, 45, 443-494.

Raudenbush, S. W., & Liu, X. F. (2003). Optimal design. Lincolnwood, IL: Scientific Software.

Rogers, E. M. (2003). Diffusion of innovations (Fourth ed.). New York: The Free Press.

Sarama, J., Clements, D. H., & Henry, J. J. (1998). Network of influences in an implementation of a mathematics curriculum innovation. International Journal of Computers for Mathematical Learning, 3, 113-148.

Sarama, J., Clements, D. H., Starkey, P., Klein, A., & Wakeley, A. (2008). Scaling up the implementation of a pre-kindergarten mathematics curriculum: Teaching for understanding with trajectories and technologies. Journal of Research on Educational Effectiveness, 1, 89-119.

Showers, B., Joyce, B., & Bennett, B. (1987). Synthesis of research on staff development: A framework for future study and a state-of-the-art analysis. Educational Leadership, 45(3), 77-87.

Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest.

Appendix B. Tables and Figures

Figure 1: Revised Network of Influences Theoretical Framework*

* Contextual variables in dotted ovals include the school (A-D), teacher (E), and child (F-H) factors. For example, child socioeconomic status, or SES (G), impacts children’s initial math knowledge (H), which influences children’s achievement (R)—an outcome variable indicated by the solid rectangle. Implementation variables in solid ovals are features that the project can encourage and support, but cannot control absolutely. For example, heavy arrows from professional development (J), to teacher knowledge (N), to implementation fidelity (O), to child achievement (R), indicate the strong effects in that path. Support from coaches (L) also has a strong effect on implementation fidelity, while other factors (J, K, M) are influential, but to a moderate degree (not all small effects are depicted). Relationships are further described in the following section.

Figure 2 Building Blocks Learning Trajectories (BBLT) Web Application

BBLT provides scalable access to the learning trajectories via descriptions, videos, and commentaries. Each aspect of the learning trajectories—developmental progressions of children’s thinking and connected instruction—are linked to the other. For example, teachers might choose the (curriculum) view and see the screen on the left, below. Clicking on a specific activity provides a description. Clicking on slides the screen over to reveal descriptions, several videos of the activity “in action,” notes on the video, and the level of thinking in the learning trajectory that activity is designed to develop, as shown below on the right. (See UBTRIAD.org for a demonstration.)

Clicking on the related developmental level, or child’s level of thinking, ringed above, switches to the view ofthat topic and that level of thinking. This likewise provides a description, video, and commentary on the developmental level—the video here is of a clinical interview task in which a child displays that level of thinking. Teachers can also study a development view, studying clinical interviews of children at each level of thinking, and, if desired, link back to activities. /