Monitoring Student and Adult Performance

A good monitoring system relies on having well-thought through and clearly defined indicators. These serve as the barometer for the district to know whether it is improving. Remember the OIP is about learning together for improvement not punishment. A good monitoring system answers the following questions:

a)What To Monitor?

b)What To Gather

c)When to Gather & Submit Data

d)How to gather data?

e)Who will gather?

f)How To Record Data

g)How To Manage Monitoring Data

h)How To Communicate Monitoring Expectations and Results

Question a) is answered by having a well-defined set of indicators. Question c), e) and g) is determined by the DLT process and schedule. Question h) should be described in the DLT communication approach of the IMM.

The following resource assists in answering questions b), d) and f) and is divided into four parts.

Part A describes ways to monitor student performance.

Part B provides ideas on what and how to monitor adult performance.

Part Cdescribes sample “look for” behaviors associated with a variety of district indicators and can be used in Part A and B.

Part D gives an option to answer question f).

Part A. Monitoring Student Performance

Monitoring student performance based on CCIP and/or SIP indicators is one of the steps in systematic problem-solving. Monitoring involves the frequent and repeated collection and analysis of student performance data. Monitoring student performance is not to be used only when time permits. Data is collected on a regular basis during the course of plan implementation. Progress monitoring provides a standardized and empirical method for evaluating the effectiveness of whether the strategies, actions and tasks implemented are improving student learning. In order to begin on-going data collection and analysis, the following must be in place and followed:

1)Have a well-defined set of knowledge and skills (academic indicators) that a student is expected to learn.

These knowledge and skills identify the focus of ongoing measurement. The knowledge and skills must be defined in terms that are concrete, observable, specific and measurable. In most cases, the target knowledge and skills should be ones that need to increase in frequency.

2)Establish measurement method(s).

The selected knowledge and skills must allow for the frequent and repeated collection of student performance data. The method(s) of measurement should be time and cost efficient, and should be sensitive to small changes in student performance over relatively short periods of time. Standardized measurement procedures should be used.

3)Describe student’s current level of functioning.

Information regarding a student's current level of functioning is helpful in setting an appropriate indicator of improvement and provides a baseline against which subsequent performance can be compared.

4)Have a well-defined strategy indicator.

It is impossible to evaluate progress without a yardstick against which to compare it. A strategy indicator provides such a yardstick by clearly describing the expected outcome of actions in terms of improved student performance. The strategy indicator should specify the behavior, the conditions under which this behavior will be exhibited and the criterion for satisfactory performance.

5)Implement actions (and tasks that put the action into motion) that will result in improved student performance.

No set of actions (and tasks) works all of the time or with every student. Plans should be developed with an expectation that they will be altered if data indicates a need to do so. At the building level, plans should clearly delineate materials and procedures to be used, as well as roles and responsibilities for collaborative team members. This may include having classroom plans in addition to a building plan.

6)Collect data regularly and visually represent the data, such as on a graph or chart.

Data must be collected in short intervals, e.g., once per week, at the building level in order to provide sufficient information for trend analysis within a month (based on the monitoring schedule the building has set) or across a quarter (based on the monitoring schedule the district has set).

Monitoring procedures generate a large quantity of data that must be systematically recorded and analyzed. A visual representation of data such as a graph or chart provides a visual depiction of expected (as described on the graph by the indicator line) and actual student performance.

7)Make course correction(s).

Course correction should allow for the systematic interpretation of performance trends with regard to progress toward the identified strategy. The process for course correction should be established prior to implementation of the actions/tasks. Course correction should include a rule for raising the strategy indicator if student progress exceeds expectations. Conversely, there also should be a rule which prompts the alteration of the actions/tasks if insufficient progress is demonstrated.

If an action must be modified, only one instructional factor at a time should be altered. If more than one factor at a time is modified, and if student performance subsequently changes, the collaborative team will not be able to determine which factor was responsible for the change in student performance. Small changes or refinements in plans should usually be attempted before major alterations in the plan are made. At the same time, however, the modifications must be ones that are judged to be sufficiently substantial to result in improved student performance.

Whenever a modification in the plan is made, this modification should be noted on the graph. A vertical line should be drawn on the graph at the point in time when the plan was changed. The collaborative team meeting notes should record the specific procedures and materials that are modified. This will assist in determining what actions/tasks should be generalized across a building/grade level/district.

B. Monitoring Adult Performance

The intent of these processes is to institutionalize observation processes that provide evidence of the degree of adult implementation of identified research-based strategies in the district and school plans. The data collected through these processes, along with the student performance data from the common formative assessments, will be the basis of facilitated conversations by collaborative teams that will lead to changes in practice.

The following describes six options. It is possible to select multiple processes. Each uses a different format for collecting data and information and reporting information.

Also included is a description of the common elements of classroom observation and limitations to them.

Classroom Observation Overview

District Walkthrough Protocol (Generic)

Learning Rounds Model

Supervisor Observation and Teacher Self-Report Using a Common Rubric Based On Instructional Design Questions

Observation Checklist Protocol

Selected Document Review

Focused Academic Review

Classroom Observation Overview

Systematic classroom observation is a quantitative method of measuring classroom behaviors from direct observations that specify both the events and behaviors that are to be observed and how they are to be recorded. Generally, the data that is collected from this procedure focuses on the frequency and duration with which specific behaviors or types of behavior occurred in the classroom. The following elements are common to most observational systems:

  • A purpose for the observation,
  • Operational definitions of all the observed behaviors,
  • Training procedures for observers,
  • A specific observational focus,
  • A setting,
  • A unit of time,
  • An observation schedule,
  • A method to record the data, and
  • A method to process and analyze data.

Although there are several types of observational procedures or techniques that have been used to examine effective teaching (e.g., charts, rating scales, checklists and narrative descriptions), the most widely used procedure or research method has been systematic classroom observation based on interactive coding systems. These interactive coding systems allow the observer to record nearly everything that students and teachers do during a given time interval. These interaction systems are very objective and typically do not require the observer to make any high inferences or judgments about the behaviors they observe in the classroom. In other words, these low-inference observational systems provide specific and identifiable behaviors that observers can easily code.

There have also been several criticisms and cautions related to the use of structured observation techniques:

(1) Theoretical Concerns: Selection of events or behaviors may not be clear to anyone except the observer or instrument developer; teaching behaviors are often viewed independent of the curricular context with which the techniques are associated, i.e. observers generally focus on isolated behaviors without concern for the preceding and subsequent behaviors that provide the context and meaning of the behavior; and it is difficult to record complex instructional behaviors.

(2) Methodological Concerns: Observer effects may occur because teachers and students are aware that their behaviors are being observed; reliability and validity of observational systems; amount of time and number of observations to obtain reliable and valid measures; and generic observations (not specific to the plan focus)

(3) Pragmatic Concerns: Extensive training and time; access to schools and classrooms to conduct observations; and misuse of classroom observation data.

The previously mentioned criticisms and limitations, however, do not necessarily detract from the value and utility of the observational method.There are examples of technology supported observation systems for conducting short, frequent, formative classroom walkthroughs. Two common systems are the iObservation™ system (Marzano) and Power Walkthroughs (McREL). Both systems, which can be customized, provide forms with content-specific language and data reports that show trends and patterns.

District Walkthrough Protocol (Generic)

A Walkthrough is: / A Walkthrough is Not:
  • A process for giving and receiving non-threatening evidence-based feedback to stimulate dialogue
  • A strategy for promoting collegiality and breaking down isolation
  • A snapshot of teaching and learning
  • An integral part of the work with improvement plans and accountability
  • A powerful tool for continuous improvement
  • A source of process data that answers the question “How are we doing?”
/
  • A “gotcha” opportunity for supervisors or peers
  • A “dog and pony show”
  • An evaluation of anyone, especially individual teachers
  • An isolated event
  • An audit

Desired results of walkthroughs

  • Frequent, formative progress
  • High-quality feedback about teaching and learning
  • Collegial dialog and collaboration
  • Data-based decision-making
  • Action planning with measurable results
  • Focused professional learning
  • On-going school improvement
  • Enhanced ownership by all

Walkthrough Process – DLT Responsibilities

Before the Walkthrough / Implementing - During the Walkthrough / After the Walkthrough
  • Introduce Walkthrough Protocol and Process to BLTs
  • Provide district-wide training/communication
  • Focus on identified curricular and instructional expectations as identified in the district plan
  • Identify configuration of walkthrough teams (e.g., DLT, other district staff, external partners, cross-building teachers and administrators)
  • Create a schedule: a minimum of one walkthrough per marking period
/
  • Walkthrough team meets with BLT to review the Walkthrough Protocol
  • BLT gives the walkthrough team an overview of current building conditions
  • Logistics are discussed, including observation form, classroom maps, scheduling and assignments
  • Conduct walkthroughs, e.g., first semester internal, second semester external, that are focused on the identified curricular and instructional expectations
  • Collect evidence using the focus questions/expectations on protocol
/
  • Walkthrough team analyzes observations with one another and generates building summary report before presenting to the BLT
  • Hold a meeting with BLT to share walkthrough summary report and plan next steps
  • BLT prepares a next steps report in response to the observation data
  • BLT shares walkthrough data and next steps with all building staff within two weeks
  • Option: DLT expands walkthroughs to include others, e.g., higher education, community, business partners, parents or a combination of participants
  • For follow-up support, DLT assigns partner schools to meet specific needs

Walkthrough Reminders for DLTs:

* Be professional * Be courteous and respectful* Build trust and maintain confidentiality; don’t gossip

*Be as unobtrusive as possible when visiting classrooms and asking questions*Enjoy the opportunity to grow professionally

Learning Rounds Model

Observation, which is often done by an individual, usually offers a snapshot of current practice and does not always lead to a better understanding of strategies which can be employed to effect large-scale practice improvement. Some districts and schools are now developing opportunities for peer observation, or observation where a colleague(s), regardless of role, conducts the observation. Learning Rounds borrows from the system of hospital ‘medical rounds and owes its inspiration to Richard Elmore of Harvard University. Using a ‘medical rounds’ model, Elmore brings groups of staff to investigate specific aspects of school improvement in order to generate strategies that can be used to improve practice. The major benefits include:

  • A focus on generating evidence-based solutions to problems/concerns faced by teachers, schools and districts around issues of system-wide improvement,
  • An opportunity for colleagues to get close to instructional practice as part of a supportive and joint learning inquiry,
  • A collaborative learning process to assist the group to explore practice and challenge assumptions, and
  • A unique and powerful form of personal PD for each colleague that will develop a self-reflective approach.

Learning Rounds is a process that is designed to deliver system-wide improvement across schools and authorities. It involves groups of staff observing and learning about and from school practice. The unique feature of this approach is that observation and inquiry are not carried out by a lone individual but by a team of colleagues who create a base of evidence around what they have seen. This evidence is descriptive only. It is not shaped by evaluative comments or value-laden points. While the collection of evidence can be done through talking to the teacher and pupils, looking at pupil work or doing a ‘walk-thru’, the most common form of inquiry is through lesson observation. It is important to note that everyone involved in this process, whether observing or being observed, are volunteers, and that the focus is on the learning of the observing group, not on the feedback given to an individual observed teacher.

Learning Rounds involves a team of staff visiting a school to explore an issue that has been identified as important, e.g., implementing research-based instructional practices. Groups of staff watch a sequence of class lessons from different teachers over part of a day. For instance, a group of three colleagues might spend the morning in a school observing segments of lessons, focusing on an agreed area of practice, such as the level of challenge through teacher questioning. If more people are involved they can be split into two groups and each group can visit a different class for the first half of the lesson and then swap round for the second half. This alternating can continue through the morning until a number of episodes of class teaching have been observed. By the end of the morning, both groups will have seen the same classes with the same teacher but at different times.

Each set of observations is followed by rich discussion in the afternoon that develops further learning. Here members of the group share their collective experiences. Because staff have observed the same lesson doesn’t mean that they have understood and interpreted the same things. It is this process of joint reflection that is so important to the learning of all involved and moves Learning Rounds away from the tradition of an observer giving an observed teacher advice and guidance, onto the equally important area of shared learning, and of next steps which are identified from agreed evidence. It is advisable to have someone facilitate to give structure to the whole process. This person:

  • Mediates the conversation
  • Guides the group to key questions
  • Does not give solutions, and
  • May introduce evidence that has been missed.

At this stage group members are not allowed to give opinions or suggestions as Elmore has found that this can lead to early disagreement and get in the way of next steps. Instead the process concentrates on getting consensus on the exact nature of the evidence in order to provide an agreed and descriptive platform to generate improvement. The visiting team then presents their evidence to the BLT. At this stage there is no evaluative comment, no suggestions as to next steps or what should be considered for change. The team’s evidence is all about what has been observed and only what has been observed.

In response to the evidence the BLT draws up a list of next steps which is presented on a return visit of the team. In this joint forum, DLT/BLT members discuss the next steps and offer evaluative opinions on the plans. This meeting is designed to be a significant driver for change in order to lay a platform for improvement and can lead to an amended plan which continues to have the engagement of the team and which can change in light