TPT Manuscript: MacIsaac & Falconer, Reform Teaching via RTOP p 1
A much-abbreviated version of this manuscript has been accepted for publication in The Physics Teacher sometime in Fall 2002 (probably November). Please do not cite it until after it has been published.
MANUSCRIPT FOR THE PHYSICS TEACHER
Reform Reform Your Classroom Teaching Instruction via the Reformed Teaching Observation Protocol (RTOP)
Dan MacIsaac
Department of Physics & Astronomy, Northern Arizona University, Flagstaff AZ 86011-6010; , 520-523-5921 (voice) 520-523-1371 (FAX)
Kathleen Falconer
Department of Physics and Astronomy, Arizona State University, Tempe, AZ 85203-1504;
Please direct correspondence regarding this manuscript to MacIsaac; authors' bios and photos are attached as a separate document.
Abstract:
This article reports recent science education research developments and explains how We report how physics teachers can use these recent science education research developments to reform refine their own teaching and improve their students’ conceptual achievement gains.
The development of the Reformed Teaching Observation Protocol (RTOP) has produced a valuable tool for reflecting upon and improving physics teaching. Instructor RTOP scores have been linked by research to found to strongly correlate with student conceptual gains in introductory science and physics courses. We discuss the background underpinnings of reformed teaching and and content of RTOP, how to use it to score constructively evaluate and critique physics teaching and to guide personal teaching improvement. Suggestions for the development of reform physics lessons are also included.
PACS Numbers: 01.40Ea,b; 0140Ga,b; 01.40H,J,K,R
Introduction: The call for reformReform physics teaching – the background from Arizona
Many pProfessional associations of scientists, mathematicians, and science and mathematics educators --such for example as the American Association for the Advancement of Science (AAAS), the National Council of Teachers of Mathematics (NCTM) and the National Research Council (NRC), have called for extensive reform in the teaching of science and mathematics (REFS 1a,b,c,d,e,f,g,h,ii,j). These reports critique US science and mathematics curricula as largely incoherent, eexcessively xcessively repetitious repetitive and unfocused – ‘a mile wide and an inch deep’ (REF 1ih, p3). Research associated withI international studies (REF 1j, p 35REF 1h, p4) has have found shown that US grade school science textbooks have many times more topics than are typical in the rest of the world, and that these American books are focused on understanding presenting simple information such as vocabulary, facts and simple equations, -- neglecting complex information synthesis, analysis and relevant application, scientific reasoning, the use of scientific tools, investigation and communication. Researchers have also found that high school physics students whose courses covered less material in greater depth did better in college courses (REF 2a). Finally, tThere here are also those who argue that the very culture of traditional physics lectures (and science lectures in general) alienates a large number of students (REF 2a, 2b) and is overdue for reform. Finally, Physics Education Research (PER) has found (REF 2c) that high school physics students who studied without commercial textbooks and whose courses covered less material in greater depth did better in their following college courses.
In 1995, the National Science Foundation funded a large five-year collaborative project called the Arizona Collaborative for Excellence in the Preparation of Teachers (ACEPT) at Arizona State University (ASU) (REF3a,b). The project goal was to better prepare K-12 teachers in science and mathematics. ASU faculty collaborated with other Arizona university faculty working on to reform the preparation of science and mathematics teachers, K-20. community colleges statewide, and local K-12 school district. Recognizing that most K-12 teachers and university faculty teach their own students as they were taught, it was ACEPT decided to "‘break the cycle"’ by reforming the freshman science and mathematics classes taken by these future pre-service teachers. Freshman science and mathematics courses would be reformedand – that is, taught via the kinds of constructivist, inquiry-based methods advocated by the AAAS, NCTM and NRC so thatthese future teachers would experience reform teaching be taught as they were expected to teach..
In order to assess whether reformed teaching was occurring at the pre-service level (and later at the K-12 level when graduates entered the teaching field), The the ACEPT group charged with assessing the project evaluation team developed a classroom observation instrument called the Reformed Teaching Observation Protocol (RTOP) tothat both measures and operationally defines reformed teaching. Appropriately enough, the instrument was called the “Reformed Teaching Observation Protocol”. Drawing upon existing observation instruments as well as the literature surrounding reform,, the RTOP was developed, fined, and validated over a period of two years. In its present form, the RTOP is a highly reliable instrument with strong predictive validity (REF 4REF 4, 5). To date, itRTOP has been used in over 400 K-20 science and mathematics classrooms to provide a precise quantitative reading of the degree to which teaching is reformed.research-grade tool, the Reform Teaching Observation Protocol (RTOP), which incorporated and extended what they felt were the best characteristics of the best of the few available rubrics for assessing science lessons. RTOP was iteratively developed by professional researchers and graduate students with master teacher input by repeatedly scoring a large collection of videotaped classroom teacher and student behaviors, refining the scored items through discussion and statistical analysis. RTOP was designed to study college-level science and mathematics classrooms to characterize ACEPT course reforms and to study the K-12 lessons taught by traditional and ACEPT-prepared teachers to see if their reform preparation led to reform K-12 teaching .By design, RTOP both operationally defines and assessesreformed teaching in the classroom – we henceforth explicitly reserve and define the term reformed teaching to mean those classroom practices that result in a high RTOP score reform (REF 4).
Insert Figure 1 about here
Research grade RTOP scores are generated by averaging several observations (at least three) taken by several observers (at least two) who are experts in teaching that subject matter and level.
After As used in In the evaluation of ACEPT, RTOP scores were found to strongly correlate with student conceptual gains (REF 5, Figure 1) showing that reformed teaching is also effective teaching. Because the correlation coefficients between RTOP and student achievement gains were so high (correlation coefficients in the 0.70 - 0.95 range were typical), it occurred to us that the items in the instrument might provide teachers as well as researchers with a window into understanding reformed teaching. Could teachers be guided to build awe use RTOP to help teachers develop a deeper understanding of the reformed nature of rtheir own eformed teaching by observing themselves and others using the RTOP? We We started to use RTOP training sessions as an explicit instructional developmentscience methods activity for pre-service teacher courses and in professional development workshops for inservice teacher professional development workshopss. This correlation is unique – while While much time and effort has been poured into reform teaching, there has been a notable lack of research linking student teacher self-knowledge and comprehension and , reform teaching and student achievement (REF 6). A review of the research lead us to believe that RTOP was appropriate for this purpose.
It may be useful, therefore, to look more carefully at the research evidence that speaks to the kind of reformed teaching that leads to strong student achievement. That literature convinced us that the RTOP encapsulates that kind of teaching.
However, there There are two relevant notable sets of research upon classroom behavior that are linked to student achievement – the physics education research of Richard Hake, and secondly, the science education research done on cooperative learning. In 1998, physicist Richard Hake (REF 7) published a large scale study of over 6000 introductory mechanics studentss (REF 7). In his study, Hake examined student conceptual gains in a series of curricula he characterized with the label ‘Interactive Engagement’:
Interactive Engagement (IE) methods… [are] …designed at least in part to promote conceptualunderstanding through interactive engagement of students in heads-on (always) and hands-on (usually) activities which yield immediate feedback through discussion with peers and or instructors… (REF7, p 65; our italics)
Hake found IE strategies produced increases in mechanics course effectiveness student achievement well beyond those produced with traditional methods. Hake indicated that the four most popular nontraditional behaviors in his IE courses studies were: 1) Collaborative Peer Instruction (REF 8a,b,c,e) (present in every IE course he studied; REF 8a,b,c), 2) Microcomputer-Based Labs (REF 8d), 3) ConcepTests (REF 8e) and 4) Modeling Instruction (REF 8f), , amongst others). Hake also repeatedly identifies curricula and methods from PER as critical to the stimulation of IE teaching methods. (REF7, p 65).
The universal role of collaborative peer activities reported by Hake in IE physics courses is itself noteworthy. Collaborative learning in small student groups has a profound number of benefits associated with it by many researchers. A large body of education research (REF 8a) reports that collaborative learning increases retention, on-task behavior, promotes achievement, positive attitudes and self-esteem and produces higher student achievement (on the order of 0.86 standard deviations above control groups across several studies).. The next three IE instructional innovations identified above by Hake all redundantly incorporate collaborative learning. The RTOP instrument is designed to constructively critique details of classroom practices including cooperative learning, interactive engagement and certain classes of PER activities and findings collectively known as Pedagogical Content Knowledge.
Henceforth, we claim that both cooperative learning and Hake’s Interactive Engagement methods are a subset of reform teaching practices better articulated and defined by RTOP.
Why and wherefore use What do Teachers get from using RTOP?
R As mentioned earlier, research shows that lesson practices scored via identified by RTOP (REF 5) are strongly correlated with student conceptual gains in introductory high school, community college and university courses in physical science, physics, mathematics, biology and biology teaching methods,. Furthermore, and this RTOP-based link between specific teaching behaviors and student achievement is both unique and compelling. The RTOP rubric is valuable because it can be readily used by physics teachers who are non-researchersneophyte as well asboth new and veteran teachers to not only generate a score their own teaching, but more importantly, to provide and feedback suggesting to acquire insight into their own teaching practices that guides their own instructional improvement and professional teaching development. With RTOP, this process the process of scoring and reflection takes about ninety minutes (assuming no more than one hour spent observing a lesson).Teachers using RTOP must work with a respected and trusted peer. RTOP scoring and initial reflection takes about ninety minutes, including one hour spent observing the lesson scored.
RTOP operationally definesclaims to be an operational definition of what reform means through as specified byspecifies a set of 25 scored, observable and scoreable behavioral itemsteachingclassroom characteristicsbehaviors or items. , and these These items themselves lead tocanItems catalyze teacher development, when each isused as a focus for reflection, discussion and debate upon observed the teaching observed.. and refinement The ensuing Constructively critical ddiscussion and debate overupon what these RTOP characteristics mean and how they manifest in one’s ownactual classroom teachingactivity underlies the development of a common language of reformed teaching grounded in personal experiences. We consider the development of a common language describing reformed teaching to be the most fruitful outcome of RTOP use by teachers – teachers are unfamiliar with reformed teaching, and all meaningful learning requires the development and refinement of precise conceptual language (REF9). RTOP items address behaviors that lie at the heart of learning science and mathematics in the classroom, unlike broader instructional rubrics such as Madeline Hunters' Elements of Effective Instruction (EEI - REF10). In some cases, these other classroom rubrics are incompatible with inquiry science learning – e.g., Hunter's rubrics for direct instruction in the classroom are centered upon teacher-directed behaviors such as "anticipatory sets" and "closure".
leads to a deeper appreciation of upon the significance of language and and the role of prior beliefs in understanding and improving reformed teaching. philosophy of reform as espoused by the RTOP rubric. As physics learning is keyed to student development and refinement of precise conceptual language, so is the learning of how to teach physics (REF9). We believe that the single most fruitful use of RTOP by classroom teachers is the development of a common language of reform teaching grounded in personal experiences. The traditional gap between talking the talk and walking the talk disappears on its own. The RTOP rubric philosophy provides insights into evaluating lessons that are specific to the intellectual process of teaching science and mathematics. Other commonly-used classroom rubrics, such as those based on Madeline Hunter’s Elements of Effective Instruction (EEI - REF10) focus on instructional organization or classroom management that are not subject specific. In some cases, these other rubrics are largely incompatible with science inquiry teaching – e.g., Hunter's rubrics for direct instruction stresses a teacher-led 'anticipatory set' and 'closure'.
T Additionally, teachers Teachers working with us have stated found that the RTOP is useful as a checklist for lesson planning purposes, in the mentoring and professional development of new or student teachers and for their own personal teaching pedagogical growth. Another commonly cited use is as justification for or inin defense of unfamiliar instructional reforms. such as An -- for example would be justifying the modeling method to administrators and parents who may be familiar with traditional instructional methods and require assistance in judging reformed instructionteaching. Many aspects of inquiry teaching challenge traditional practice and teachers tell us that RTOP helps validate and refocus their own teaching practices journey into reform..
An Overview of RTOP
ObtainingGetting your own RTOP score usingvia the Instrument
To obtain an RTOP score of one of your own lessons, 1) download the RTOP Training Manual (REF 4) and print a copy for yourself and a teaching colleague whom you trust and respect, ideally familiar with teaching your subject. 2) You and a colleague should read and discuss the instrument, then 3) arrange for your colleague to visit your class to observe and RTOP an hour lesson. 4) While your colleague observes your class, have a student or aide videotape your lesson. 5) RTOP this videotape yourself, before discussing your colleague's RTOP score of your lesson. 6) Reciprocate -- perform an RTOP observation on your colleague in turn. This will provide more needed classroom observation material for discussion and genuine meaning in this experience for both of you. 7) Meet with your colleague to discuss and attempt to reconcile the scores on each of the twenty-five items. Inevitably, you will disagree with your colleague. Use the differences as a focus for re-examining your own teaching practices.
T
The In addition to collecting information about background and contextual activity descriptors, t he RTOP instrument is divided into five sectionsRTOP instrument items are divided into five sections: the collection of background and contextual activity descriptors,(1) lesson design and implementation information, (2) propositional content knowledge, (prepositionaland (3) procedural content knowledge,)and (4) classroom culture (student-student and student-teachercommunicative interactions) and (5) classroom culture (student-teacher relationships). These last three major five sections include the twenty-five scored items. In general, the twenty-five items can be summarized for physics lessons as follows:
observable items scored from 0 - 4 as follows:
0the behavior never occurred
1the behavior occurred at least once
2occurred more than once; very loosely describes the lesson
3a frequent behavior or fairly descriptive of the lesson
4pervasive or extremely descriptive of the lesson
-- where the exact details of the intermediate scores differ for each of the twenty-five items and have been rigorously defined by researchers. Research grade RTOP scores are generated by averaging several observations (at least three) taken by several observers (at least two) who are experts in teaching that subject matter and level. For your use, when in doubt scoring err on the side of a lower score. If you feel uncomfortable with a five step gradation, try assigning only scores of 0, 2 or 4 (absent, sometimes present, always) for an item. If you didn’t directly observe an item it scores as zero (do not make any inferences without training).