RESEARCH REPORT of the Classroom Assessment as a basis for Teacher Change (CATCH) project[1]
Freudenthal Institute, The Netherlands
University of Wisconsin–Madison
Co-Principal Investigators Jan de Lange
Thomas A. Romberg
Project Coordinators
Utrecht, The Netherlands Truus Dekker
Madison, Wisconsin David Webb
Researchers
Utrecht, The Netherlands Mieke Abels
Corine van den Boer
Els Feijs
Henk van der Kooy
Nanda Querelle
Martin van Reeuwijk
Monica Wijers
Secretary Patricia Boustany
Research Report CATCH August 2003
Truus Dekker & Els Feijs
with contributions from Martin van Reeuwijk and the CATCH Team
Abstract
The main focus of the Classroom Assessment as a Basis for Teacher Change (CATCH) project was to develop and test a program of professional development that seeks to bring about fundamental changes in teachers’ instruction by helping them change their formative assessment practices. Teachers from Philadelphia, PA, and S. Milwaukee, WI, were involved in the project, with four experienced teachers from Providence, RI, lending assistance in commenting on written materials and serving as support for new project teachers. In addition, select district administrators at both research sites participated in the project by facilitating monthly meetings, coordinating summer institutes for professional development, and securing necessary cost-share for district-wide dissemination of assessment principles and practices. The intervention studied in this project involved professional development activities to achieve teachers’ classroom assessment practices that support teaching and learning for understanding. Initial observations of CATCH teachers at both research sites found that most teachers had retained conventional assessment practices even though they were using various reform-based mathematics curricula.
Evidence from successive rounds of teacher interviews and classroom observations revealed a broadening of teachers’ conceptions of classroom assessment and an increase in teachers’ use of assessment tasks that provide students opportunities to demonstrate varying degrees of mathematical understanding. Participating teachers value student thinking, have grown in their effectiveness in designing assessment tasks, and are more adept at eliciting evidence from students and identifying student misconceptions. Teachers are actively involved in designing and using formative assessment tools in their own classrooms and interpreting students’ mathematical representations. They presented their findings to colleagues at CATCH summer institutes. Teachers and school administrators also have been examining how changes in classroom assessment practices affect student achievement. In general, student achievement scores on standardized tests improved and students taught by CATCH teachers had higher scores than those taught by non-CATCH teachers.
With respect to dissemination of the CATCH professional development ideas and scaling up, evidence was found of ideas traveling to other curriculum areas as well as to larger groups of teachers and other grade levels. Appropriate organizational structures have also enabled travel on a district level.
Research Report CATCH August 2003
Truus Dekker & Els Feijs
with contributions from Martin van Reeuwijk and the CATCH Team Abstract
1
Research questions answered
1. What professional development materials will be required to disseminate principles for improving formative assessment across a wide range of schools?
Several professional development materials were designed for this project.
o De Lange, J. (1999). Framework for classroom assessment in mathematics.
Unpublished manuscript. Madison, WI: National Center for Improving Student Learning and Achievement in Mathematics and Science. (Available at www.fi.uu.nl/catch/products/framework)
The design model presented in this document forms the basis of the GAP book, (see next product), which is designed to elaborate theoretical principles of the framework in a practical way for teachers, with many examples of item formats, problem contexts, and techniques to assess mathematical competencies. Work on this framework started during the previous RAP project and revisions took place during the CATCH project.
o Dekker, T., & Querelle, N. (2002). Great assessment problems. Working manuscript. Madison, WI: National Center for Improving Student Learning and Achievement in Mathematics and Science. (Available at www.fi.uu.nl/catch/products/GAP_book/intro.html)
o Abels, M. (2003). Elements for professional development (CATCH professional development guide).Working manuscript. Madison, WI: National Center for Improving Student Learning and Achievement in Mathematics and Science.
o Romberg, T. A. (ed., in press). Insight stories: Assessing middle school mathematics. This book was published during this period though it describes results from the RAP project.
An assessment tool, AssessMath! (Cappo, De Lange & Romberg,1999), developed in a separate project, was also used.
Analysis of the interviews offered evidence of ideas traveling to other curriculum areas as well as to larger groups of teachers and other grade levels. Appropriate organizational structures have also enabled travel on a district level. CATCH lead teachers contributed to both. The lead teachers also contributed to a number of conferences and seminars. CATCH lead teachers in S.Milwaukee were included in the writing of the district mathematics framework and the design of open-ended items used on district-wide assessments. In Philadelphia a large number of teachers became involved in the CATCH professional development program. The district has now submitted a proposal to expand a follow up of the CATCH program to K-12 teachers in mathematics as well as science.
Other districts have shown interest in the CATCH professional development program.
2. What support do school personnel and teachers in various school contexts, who are adapting these principles to local conditions, need to ensure that changes in formative assessment are sustained?
A CATCH web site (www.fi.uu.nl/catch) was designed by Mieke Abels and Martin van Reeuwijk. This web site was intended for continued teacher support and exchange of ideas among participants of the project. All chapters of the GAP book as well as the Framework have been posted on the web site for participants to provide comments and submit supplementary information. Tasks and test items were published for teachers to use in their own classrooms. For each research site, a separate directory was reserved on the web site to publish minutes of monthly meetings and related reports.
Four times during the project, a CATCH newsletter composed by Monica Wijers, Corine van den Boer and Mieke Abels, with contributions from the CATCH team as well as from project participants was published on the web site and mailed to all participants.
The analysis of the interviews showed, however, that the most outstanding source of support was the frequent personal contact with colleagues, whether through professional meetings or through informal contacts. Participants also highly valued the information received at the initial professional development seminar in Utrecht, and the contact with members of the CATCH team in person through regular site visits which to teachers’ delight included opportunities to attend and discuss their classes. In the interviews teachers furthermore mentioned the support of different types of materials they received and indicated how important the organization of their own local summer institutes had been in their professional development.
3. How do teachers make decisions about what assessment instruments to use, when to use them, and which reasons motivate their choices?
In response to the No Child Left Behind Act, the South Milwaukee Math Leadership Team has been developing mathematics assessments to be used district wide in Grades 3-8. These tests were designed as diagnostic tools to monitor grade-by-grade student performance on district content standards and were based, in part, on assessment design principles from the CATCH project.
The S.Milwaukee teachers have also been collaborating with members of the Math Leadership Team (MLT) to review district content standards and benchmarks and to identify mathematics content that should be reflected in each assessment. Assessment items were selected and adapted from such resources as public release items from the Third International Mathematics and Science Study and AssessMath. The MLT also designed their own tasks as needed, including items at various reasoning levels to assess a wider range of student competencies.
From March through May 2003, assessments were administered and scored by the MLT, giving teachers student performance data that reflected the district content standards. With this new achievement data in hand, district and school administrators requested additional information from the Math Leadership Team: What are the proficiency levels for each assessment? Can these assessments be used to identify students for advanced math courses or instructional interventions? Such requests pressed teachers to discuss and define what it meant to be “proficient” in mathematics.
By engaging in the process of defining student proficiency, teachers came away with an improved understanding of the role of professional judgments in making assessment-driven decisions. However, teachers also articulated several dilemmas in using an end-of-year assessment to identify student proficiency. Were the assessments fair representations of the mathematics content taught at each grade level? To what extent did these results, from a 50-min written assessment, accurately measure student knowledge and understanding? Are there other indicators that should be used district-wide? To what extent should we take into account (in the design and scoring of assessments) students’ differing opportunities to learn (e.g., whether fractions, taught in 5th grade, were covered before or after the assessment)?
4. How do teachers’ assessment practices change as a result of their participation in this professional development program?
We found changes in participating teachers’ attitude toward formative assessment as well as changes in their classroom practices as a result of their participation in the professional development program offered to them. A change in attitude was accomplished for all teachers shortly after the initial professional development program was held; a change in classroom practice appears to take more time but was nevertheless reported by all participants within a year’s time. The level of change was sustained to a great extent throughout the remaining project time, with the exception of teachers in Philadelphia who suffered from rather dramatic changes in their teaching circumstances. The most striking changes in teachers’ attitude toward formative assessment were related to the assessment pyramid, which was a crucial element of the CATCH professional development program designed to support changes in teachers’ formative assessment practices. Our findings indicate that ideas related to instructional embedded assessment and the critiquing of teachers’ own assessment instruments and trying other assessment formats were more easily implemented, but nevertheless over time there was a growing number of teachers managing implementation of assessment pyramid related ideas.
5. How are changes in teachers’ assessment practices reflected in their students’ achievement?
Because of the valid control group in Philadelphia for the school year 2001-2002 and because of the occurrence of the same trend in both school districts we carefully conclude that the scores on standardized tests of students taught by teachers involved in the CATCH project went up considerably during this project.
Research Report CATCH August 2003
Truus Dekker & Els Feijs
with contributions from Martin van Reeuwijk and the CATCH Team Research Questions Answered
4
Project Description
Conceptual Background
CATCH was a continuation of the Research on Assessment Practices (RAP) study group. The evidence collected through this project showed that many teachers have limited understanding of formative assessment practices and, thus, provide students with incomplete information about their progress (Romberg, 1999). Moreover, in the RAP study we found that teachers needed technical assistance with assessment design and that they sought tools and methods to further develop their capacity to assess student learning.
In line with the research reported by Black and Wiliam (1998) we agree that mathematics instruction that promotes understanding is highly dependent on appropriate formative assessment practices and tools. Hypothetical assessment trajectories (HATs) require an availability of student responses that can be observed and heard so that teachers and students can reflect, infer, and act on interpretations and solution strategies to a problem situation. We view HATs as a loosely sequenced subset of benchmark evidence for student learning with understanding, bounded by practical means by which teachers can reasonably assess individual and collective learning within a classroom setting through formal, informal, and instructional assessment practices. Using learning and assessment trajectories as a developmental framework requires teachers to conceptualize student learning as a network of desired developmental paths, accessible and achievable through appropriate sequencing of learning opportunities.
Our previous research led us to assert that formative assessment is quite complex. Emerging research of teachers’ instructionally embedded assessment practices has revealed that teachers’ concerns with assessment are not strictly an issue of resolving problems related to formal assessment. Teachers also are interested in exploring the use, for assessment purposes, of their students’ mathematical representations elicited in a variety of instructional contexts (Webb, 2001). Some teachers have found that restructuring and redefining their grading system according to content goals can support more purposeful instruction and motivate further study of the ways students represent and communicate their understanding of mathematics during instruction (Her & Webb, in press; Webb, Romberg, Burrill, & Ford, in press). This suggests that when teachers explore and reflect on the ways they formally assess student understanding, their inquiry is likely to influence the instructional activities they choose, the questions they ask students, and how they facilitate classroom discourse. Our intent was to bring these formative assessment processes to the fore. Developmental research suggests that it is unrealistic to expect teachers to become instant assessment designers and experts (De Lange, 1987). Therefore, informed choices need to be made.
It is on the basis of these past experiences that we assumed that changing the assessment practices of teachers already using reform curricula could play an important part to enhance “teaching and assessing for understanding”. By posing questions and using test items that evoke mathematical reasoning and generalizing and by asking students to choose their own mathematical tools to solve a problem, it becomes clear what students are able to do instead of which facts, standard algorithms and definitions they know. If the process is as important as the product, if strategies used by students are important, it becomes necessaryimportant for the teacher to listen carefully to what students say in class and assess student work more closely. Teachers use the information gathered this way to guide instruction. Textbooks do not always provide good problems. So a crucialn important step for teachers to be taken is to adapt questions posed in the curriculum or to enhance their own assessment problems. Then the design of balanced assessments is being discussed, using problems at different competency levels. Assessing becomes a continuous process, an integrated part of the teaching and learning process instead of something from outside, interrupting this process but nevertheless inevitable. We feel that this results in a more “student centered instructional environment, more likely to improve student achievement” (Bransford, Brown & Cocking, 1999).
Professional development: uUnderlying ideas, sStrategy and mMaterials