Roles and Responsibilities of Trainers

a. Roles and Responsibilities of Trainers

This explains the role a trainer of scorers plays before and during scoring, including the calibration, double scoring, and failing (or close to failing)Teaching Event policies.

Before training

Locate copies of the current benchmarks and calibration Teaching Events and make sure that the videos can be viewed on the equipment to be used. If there are problems with viewing the videos, consult local technical staff or e-mail Kendyll at . Do this well in advance of the training
Review the benchmarks and the training protocol prior to training

Before scoring

Deliver training according to PACT guidelines
Review completed scoring forms for calibration Teaching Event(s) to determine whether scorers have calibrated. To calibrate, a scorer’s scores, compared to the previously determined scores:

Must result in the same pass/fail decision
Must include at least six exact matches
Cannot include any scores that are two away from the pre-determined score.
In addition, the evidence should support the score chosen.

For scorers who fail to calibrate, provide feedback and remediation. If they were close to calibrating, then do a close read of their completed scoring form for the next Teaching Event. If they were not close, ask them to complete another calibration Teaching Event and review for success. Repeat as needed. It is up to the program/trainer to determine the point at which to stop recalibration if a scorer repeatedly fails to calibrate.
Unless your program assigns Teaching Events to scorers centrally, make preliminary assignments of Teaching Events to scorers, designating these scorers as first scorers.
Unless your program has a central process for selecting Teaching Events to be double scored, select a random sample of 10 % of the Teaching Events for double scoring and assign to scorers, ensuring distribution across scorers. Designate the second set of scorers as second scorers.

During scoring

Keep track of the receipt of scored Teaching Events from scorers.
Review the evidence from the first completed scoring form for each scorer, prioritizing new scorers and scorers who did not calibrate on the first attempt. If the evidence does not support a score, then review the evidence and the big ideas in that scoring rubric with the scorer.
Compare the two sets of scores for Teaching Events that are double scored. Read and rescore rubrics where scores are two or more apart. For scorers whose score(s) are two apart from your score(s), give them feedback on their strengths and weaknesses in scoring and go over the discrepant scores comparing their evidence and your evidence. As possible, continue to monitor completed scoring forms. If a scorer is consistently higher (or lower) in double scoring, then consider whether recalibration is needed.
Identify 1) failing Teaching Events, 2) any Teaching Event receiving one or more scores of Level 1, and 2) any other Teaching Event that scorers might identify as problematic and ensure rescoring of these Teaching Events. After rescoring, compare the scores. If scores are two or more points apart or the two scorers disagree on whether the candidate passes or fails a task, rescore the rubrics for which scores are discrepant.

Last edited 6/1/09