An End-Of-Project External Evaluation Report

1

An End-of-Project External Evaluation Report

Improving the Quality of Arizona Teachers of

Physics, Chemistry, Physical Science and Mathematics

Robert J. Culbertson, Principal Investigator

ArizonaStateUniversity

Reported by the External Evaluator:

Rose Shaw, Ph.D.

Metrica

1703 36th Avenue Court

Greeley, CO80634-2807

970.330.3161

Report Date: July 29, 2008

ASU ITQ Project – Year 2 – Final Report – July 2008

1

Table of Contents

Introduction to the Project / Page 1
Project Impact: Peer Leaders / Page 1
Project Impact: Teachers from the Partner District / Page 2
Project Impact: Becoming More Highly Qualified / Page 3
Summative Evaluation: RTOP Observations Over Time / Page 4
2007-08 RTOP Observation Narrative Report / Page 8
Project Strengths / Page 20
Project Challenges / Page 21
Internal Evaluation Process / Page 21
Evaluation Instruments and Protocols / Page 21
External Evaluation Activities / Page 22
Interview of Patricia Burr / Page 22
PHS 534: Two Teachers’ Reflections / Page 23
Use of the RTOP by Teachers / Page 24
External Evaluator’s Comments / Page 25
Appendix A: Jane Jackson (6/24/08 and 6/25/08) Emails / Page 26
Appendix B: Challenges to Teaching Inquiry Science / Page 28

ASU ITQ Project – Year 2 – Final Report – July 2008

1

Introduction to the Project: Improving the Quality of AZ Teachers

The Arizona State University Department of Physics and Astronomy and its chief high-need Phoenix Union High School District (PUHSD) formed a partnership to implement Modeling Instruction in High School Physics, which was designated exemplary in 2000 by the U.S. Department of Education’s Mathematics and Science Education Expert Panel, to improve the quality of participating teachers’ physics, chemistry, physical science and mathematics pedagogy and Arizona standards-based content. Teachers will participate each summer for two years in summer modeling workshops and other content courses in the physical sciences with mathematical modeling, and three full-day follow-up sessions each year. The physical sciences, technology and mathematics are integrated in this professional development with thematic strands in scientific and mathematical modeling and the use of technology as a scientific tool. Previous evaluations of Modeling Instruction verified that the science courses taught by the participating teachers will be more rigorous resulting in improved student learning in science.

Project Impact: Peer Leaders

One of the impacts of this project was that it provided the opportunity for teachers who participate in the workshops and courses to become Modeling Instruction leaders. Peer leaders, teachers who are invited by the Modeling Instruction Program co-lead workshops with experienced modeling workshop leaders. Peer leaders of four workshops CHM 594, PHS 530, PHS 542 and PHS 594 highlighted their strengths in lesson planning and implementation. The CHM 594 peer leader wrote, “I’ve taught chemistry for over 25 years. I’ve worked hard to find ways to organize the high school course around models instead of topics, as is most curricula today. I am able to help teachers recognize that teaching by telling is ineffective and can help they find ways to get their students to develop a better conceptual understanding of chemistry. I have a good grasp of the current chemistry education research and help the teachers in the workshop recognize I conceptions and how to deal with them.” All four peer leaders reported having had a high quality experience as a peer leader working with an experienced workshop leader. Two of them commented on their experience:

• I have had the opportunity to work with several dedicated and talented colleagues who have not only helped me run workshops, but have also developed the skills to lead workshops in their own right.
• I learned a lot as a new workshop leader—about the issues teacher in other states and countries have and of the apprehensions that they feel about implementing in their classrooms.

They all provided anecdotal evidence pertaining to the impact of the workshop on the participating teachers:

• Teachers tell us that the workshop is a “transformative experience”; i.e., that they cannot go back to teaching in the manner that they have been using prior to the workshop.
• I received an email from one of the participants who is implementing student blogs, as a way of connecting student discourse outside the class. This was something she learned in our workshop. Another teacher from the workshop returned to ASU this week to attend the AMTA (American Modeling Teachers Association) meeting held there. He was impressed enough with modeling to want to drive back, from Southern California, to learn more and collaborate and socialize with other modelers.
• No teacher leaves a modeling workshop the same as they came. One phrase we constantly used while looking at relationships between two quantities was “Are they ‘the same’, ‘not the same’ or ‘the same but not the same’?” My most special moment I in the third week after we relaxed and got to know each other: At lunch a Jewish woman from NY/Phoenix and a Muslim woman from Kuwait were sitting like sisters and discussing differences and similarities in being women in their religions. God was smiling: “Same but not the same.” I am crying as I think about this. I was “the leader” in this workshop yet grew the most.
• Each summer I see teacher who took my workshop the previous year. The fact that they are coming back for another class should indicate that the modeling workshops have a positive impact on teachers. I also get a chance to hear about their successes in the classroom. They become firm believers in the modeling method.

Project Impact: Teachers from the Partner District

Four teachers from the partner school district who participated in the ITQ project during its two years summarized the impact of their participation.

• The impact of the modeling courses has been enormous. My background, like many physics teachers, was NOT in physics. When I was told that I would be taking over the general physics classes at my school, I immediately started calling around my district for help from other physics teachers – I fully intended to steal, beg, or borrow anything I could get. Lucky for me what I got was advice to take the first of the modeling courses. I used a lot of “cookbook” labs when I taught chemistry. Now, I can't imagine doing that – everything is inquiry. Everything is discussed, debated, turned upside down and inside out. The students tend to resist at first; they're much too used to sitting passively and taking notes. But once they buy in, they love it. I love it. The students get much more out of the content and I get more enjoyment out of teaching.
• It has motivated me to refine and continue to improve the way I work with students to provide them with opportunities to learn and understand the concepts they are asked to master.
• Modeling has been a huge impact and continues to be today.I am making progress each year in getting my students to think (my overall goal when I started in 2006). Modeling allows me to use the different strengths of students when they are grouped and help them help each other to understand the topics we are covering. I’m looking forward to this year and with the new ideas I want to try out in class. Last year was a bit harder to try since we had just received a new book and the school asked us to try and stay on the same page on testing and other assignments. That made it difficult since I was not familiar with the new book.
• The impact of modeling has been tremendous. It has influenced me to put the onus on the student to think on how to solve problems (mathematically and physically). It also highlights misconceptions that students have and constantly addresses them. I especially like the representations offered by modeling that helps students visually what is happening. Although, I think that modeling is lacking on the writing component of science and some sort of reference text.

The impact of the project is also reflected in the motivation the Modeling Program provides for teachers to continue with this professional development model. Three teachers talked about their motivations for continuing with the modeling workshops during the two years of the project:

• The primary reason is that I need them for my MNS degree. However, I would have taken the modeling classes even if I wasn't working on the degree; they provide a fantastic combination of physics content and teaching methodology.
• I have been motivated by the quality of the classes and the different ideas that I can use and incorporate into my own classes. And the price is also something I can afford, being a new teacher in this area; otherwise I could not take these classes.
• I am motivated by the innovated ways of teaching physics and as well as the content review. I took Mechanics and Optics in the modeling classes. A nice byproduct is the interaction of fellow modelers, too.

Project Impact: Becoming More Highly Qualified

Three of the four teachers who teach in the partner school district and were not highly qualified at the beginning of the project became highly qualified. One of the teachers referred to becoming more highly effective even though she was highly qualified.

• Becoming “highly qualified” through the modeling courses was MUCH better for me than if I'd taken more “traditional” content classes. These classes provided the content AND the methodology in ways that really pushed me to think and process. This will translate much better into the classroom.
• I am going to take the physics test later this year and hopefully become highly qualified there.
• I was already highly qualified in physics and mathematics. But it sure did help a lot to be in the courses and workshops.

Just as they had during the interviews in 2006, teachers were asked about how highly they implement seven components of modeling. Rating means and standard deviations are displayed below. These are self-reported, non-rubric determined ratings that should not be overly interpreted. One of the teachers who recorded 2008 ratings of 3.5 on modeling strategies and Socratic questioning wrote, “While I tried to make full use of all of the above, I'm still working on how well/effective that implementation is. I scored myself lower on the last two (and thus the strategies category as well) as those are the areas that I struggle most with. However, I do recognize the issues and am working to improve.”

These teachers reported being more effective at implementing all aspects of Modeling as a result of having participated in the project for two years.

Summative Evaluation: RTOP Observations over Time

The Reformed Teaching Observation Protocol (RTOP) was developed as an observation instrument to provide a standardized means for detecting the degree to which K-20 classroom instruction in mathematics or science is reformed.[1] The protocol was used to assess levels of implementation of reformed science instruction by 10 ITQ project teachers from the partner district, Phoenix Union High School District (PUHSD). To ensure confidentiality, after the observations, no information about the observations was shared with PUHSD or the teachers. The External Evaluation contacted Deedee Falls of PUHSD about the observations; teachers and their principals approved the observations. The following table summarizes when the ten teachers were observed. Note that four teachers (A, B, C and D) were observed all three times.

Table 1: PUHSD chemistry, physics and physical science teachers observed during project years 1 and 2
Teacher by Code / Spring 2007 Pre- / Fall 2007
Post- / Spring 2008 Post-post
A / X / X / X
B / X / X / X
C / X / X / X
D / X / X / X
E / X / X
F / X
G / X / X
H / X
I / X / X
J / X

Summative RTOP Evaluation: Mean Ratings

The highest possible total score on the RTOP is 100. Ratings are 0 to 4 (0-never occurred; 4-very descriptive) for 25 dimensions of effective instruction grouped into five categories: Lesson Design and Implementation, Propositional Knowledge, Procedural Knowledge, Communicative Interactions and Student/Teacher Relationships. The following bar graph displays the mean total scores for pre-, post- and post-post time frames. The mean ratings for each dimension for each of the three observations are displayed in the tables that follow the bar graph. The means of the subtotals for each of the five categories are also displayed in the five tables.

Lesson Design and Implementation / Pre-
Mean / Post-
Mean / Post-post
Mean
1) The instructional strategies and activities respected students’ prior knowledge and the preconceptions inherent therein. / 1.7 / 3.0 / 3.7
2) The lesson was designed to engage students as members of a learning community. / 1.7 / 3.3 / 3.0
3) In this lesson, student exploration preceded formal presentation. / 1.0 / 2.5 / 2.2
4) This lesson encouraged students to seek and value alternative modes of investigation or of problem solving. / 1.6 / 1.4 / 2.0
5) The focus and direction of the lesson was often determined by ideas originating with students. / 1.1 / 2.1 / 3.0
SUBTOTAL / 7.1 / 12.3 / 13.8
Content: Propositional Knowledge / Pre-
Mean / Post-
Mean / Post-post
Mean
6) The lesson involved fundamental concepts of the subject. / 2.3 / 4.0 / 3.3
7) The lesson promoted strongly coherent conceptual understanding. / 1.4 / 1.9 / 3.0
8) The teacher had a solid grasp of the subject matter content inherent in the lesson. / 2.6 / 4.0 / 4.0
9) Elements of abstraction (i.e., symbolic representations, theory building) were encouraged when it was important to do so. / 1.9 / 2.5 / 3.0
10) Connections with other content disciplines and/or real world phenomena were explored and valued. / 1.1 / 2.4 / 2.3
SUBTOTAL / 9.3 / 14.7 / 15.7
Content: Procedural Knowledge / Pre-
Mean / Post-
Mean / Post-post
Mean
11) Students used a variety of means (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent phenomena. / 1.6 / 1.6 / 2.2
12) Students made predictions, estimations and/or hypotheses and devised means for testing them. / 0.3 / 1.5 / 2.5
13) Students were actively engaged in thought-provoking activity that often involved the critical assessment of procedures. / 2.1 / 2.4 / 3.2
14) Students were reflective about their learning. / 1.9 / 2.7 / 2.8
15) Intellectual rigor, constructive criticism and the challenging of ideas were valued. / 1.3 / 2.6 / 2.7
SUBTOTAL / 7.1 / 10.9 / 13.3
Classroom Culture: Communicative Interactions / Pre-
Mean / Post-
Mean / Post-post
Mean
16) Students were involved in the communication of their ideas to others using a variety of means and media. / 1.0 / 2.1 / 2.7
17) The teacher’s questions triggered divergent modes of thinking. / 1.7 / 1.7 / 2.5
18) There was a high proportion of student talk and a significant amount of it occurred between and among students. / 2.0 / 3.0 / 2.8
19) Student questions and comments often determined the focus and direction of classroom discourse. / 1.6 / 2.6 / 2.5
20) There was a climate of respect for what others had to say. / 1.3 / 3.0 / 3.0
SUBTOTAL / 7.6 / 12.5 / 13.5
Classroom Culture: Student/Teacher Relationships / Pre-
Mean / Post-
Mean / Post-post
Mean
21) Active participation of students was encouraged and valued. / 2.0 / 3.6 / 3.5
22) Students were encouraged to generate conjectures, alternative solution strategies, and ways of interpreting evidence. / 1.0 / 1.5 / 2.5
23) In general the teacher was patient with students. / 1.6 / 3.5 / 3.8
24) The teacher acted as a resource person, working to support and enhance student investigations. / 1.9 / 3.1 / 3.3
25) The metaphor “teacher as listener” was very characteristic of this classroom. / 1.4 / 2.7 / 3.0
SUBTOTAL / 7.9 / 14.5 / 16.2

RTOP Ratings for Teachers A, B, C and D from Pre to Post-post

Teachers coded as A, B, C and D were observed pre, post and post-post. The total RTOP scores for these three teachers are displayed in the following bar graph. Total RTOP scores increased significantly over time (repeated measures AOV, F = 13.59, p = .0059).

Mean pre, post and post-post scores for each of the category subtotals are displayed in Table 2 for the group consisting of the four teachers who were observed all three times. All five mean subtotal (highest possible score was 20) and the total scores (highest possible score was 100) increased significantly from pre to post-post.

Table 2: Mean RTOP Category Subtotals and Totals of Four Teachers Pre, Post and Post-post
Category / Pre / Post / Post-post
Lesson Design and Implementation / 8.7 / 14.3 / 14.5
Content: Propositional Knowledge / 10.0 / 14.3 / 16.0
Content: Procedural Knowledge / 8.0 / 13.0 / 14.0
Classroom Culture: Communicative Interactions / 8.5 / 14.7 / 14.3
Classroom Culture: Student/Teacher Relationships / 9.0 / 16.7 / 17.0
TOTAL SCORE / 44.3 / 73.0 / 75.7

Conclusions

RTOP ratings recorded by certified RTOP observations of project teachers during project years 1 and 2 indicated that project participation increased the use of effective teaching strategies.

External Evaluator’s Note

We are going to submit a manuscript of this summative study to Physics Teacher Online. The writing of it is in process.

2007-08 RTOP Observation Narrative Report

This is a narrative report of the spring 2008 RTOP observations. Quantitative analyses were included in the summative RTOP evaluation report. Teachers in this report are identified by the codes in the summative external evaluation RTOP report: Teachers A, B, C and D were observed three times during this project and teachers E and I were observed twice.

Teacher “A”

Class began with bell work on the overhead projector:

1. Draw particle diagram representing 5 molecules of gas A in a container and 10 molecules of gas B in another container the same size.
2. How does the pressure in the 2nd container compare to that in the first?
3. Represent a 3rd container that contains all the molecules in the other 2 containers (5 of A and 10 of B).
4. Predict how the pressure of the mixture of gases would compare to that in each of the other containers.

The teacher set a timer for 10 minutes. Some students chatted about the bell work questions and others about social stuff. Most students appeared to be attempting the bell work. “We did these particle diagrams so long ago.” The teacher stood at the front of the room responding occasionally questions from students—most of these were of a logistical nature. The teacher warned them when there was two minutes remaining.

When the timer went off the teacher got out a stamp pad and asked them to put their homework from the previous night on their desk. The teacher went around the room stamping papers and then began to go over the bell work. The teacher raised the projector screen to reveal three containers drawn on the board underneath and then arranged big plastic molecule stickers on the board, asking about pressure and number of collisions, and how much greater will the pressure of B be than of A. Triadic dialogue ensues, eventually arriving at Dalton’s law of partial pressures – the pressure of two combined gasses in a container is the sum of the partial pressures of these gases. “Can we say what percentage of the pressure is due to A and what is due to B?”