Mentors’ written lesson appraisals: do pre-service teachers’ perceptions of content match with reality?

Roger Lock, Allan Soares and Julie Foster

Paper presented at the British Educational Research Association Annual Conference, University of Warwick, 6-9 September 2006

Introduction

The rationale for focusing on written lesson appraisals is that most post-lesson feedback received by pre-service teachers is verbal and there appears to be limited information in the literature about any post-lesson written feedback (Spear et al,. 1997; Bunton et al., 2002). Bunton et al., (2002) see written lesson feedback as a ‘permanent record, unlike speech which may be forgotten or inaccurately heard and remembered’ (p. 233). In busy teachers’ lives the written feedback could also be considered during, after and in place of any verbal feedback.What has been identified in the limited literature about post-lesson written feedback is the tendency for mentors to comment on class management issuesat the expense of subject related pedagogical issues. Spear et al.(1997) reported, in their study of ‘the form and substance of the written feedback’ (p270) provided for primary student teachers, that commentsfocused on control and discipline, as did Lock (2002) for secondary biology pre-service teachers. More recently Lock and Soares (2005) reported on a study where pre-service biology and chemistry teachers were mentored by two kinds of science mentors; one a generic mentor termed the science mentor and the other which had specific training to concentrate on subject related pedagogical issues called the PhySEP mentor. The pre-service biology and chemistry teachers, in their sample of 29 teachers, perceived marked differences in their written lesson appraisals. The science mentors’ post-lesson written feedback were seen as concentrating primarily on class management issues while the PhySEP mentors were more focussed on subject related pedagogical issues.

In this paper we question whether the pre-service teachers’ perceptions of content match the reality of written lesson appraisals. In addressing this question we describe, in note format, the context in which the written lesson appraisals were obtained and the tool for analysing these appraisals. We also present, in note format, our findings from the content analysis of mentors’ written appraisals (n=30) and explore how this matches the pre-service teachers’ perceptions reported previously (Lock and Soares, 2005). Finally we consider the implications for future practice.

Context of written lesson appraisals

•WLAs of 15 students, drawn at random from a pool of 29

• 2 WLAs for each student

- midway through school placement

- One from each type of mentor

Research tool

A. Structure

Three categories of phrases were used which arose from the student response to a questionnaire given to students, asking for the most frequently addressed issues.

These were:

1. Topic specific e.g.

- Resistors work by slowing down the flow of electrons

- Good for asking ‘does conduction move heat up

2. Control and discipline e.g.

-Very few were paying attention when you showed the answer

- Class settled well

3. Generic e.g.

- Use OHT’s where necessary instead of writing on board

- Pupils copy down aim of lesson

In the literature, control and discipline and generic are usually referred to as generic pedgagogy.

1

B. Reliability

Cohen’s Kappa (k)

Range: Fair 0.4-0.6 Good 0.6-0.7 Excellent above 0.7

Intra-rater reliability
(Rater A) / Inter-rater reliability
Candidate no. / k / Rater / k
1 / 0.7 / A v B / 0.5
9 / 1.0 / A v C / 0.4
12 / 0.8 / B v C / 0.5

Used Cohen’s kappa to test reliability. Some research takes inter-reliability as the touchstone and this is seen only as ‘fair’.

Findings

  1. Number of phrases

Candidate
number / PhySEP / Science
1 / 44 / 94
2 / 51 / 69
3 / 59 / 46 / PhySEP / Science
4 / 25 / 33 / Mean / 50.87 / 62.67
5 / 50 / 50 / Standard Deviation / 30.04 / 24.91
6 / 52 / 30 / Probability / 0.25
7 / 28 / 24
8 / 51 / 64
9 / 38 / 80
10 / 23 / 98
11 / 34 / 56
12 / 41 / 89
13 / 64 / 88
14 / 150 / 39
15 / 53 / 80

Higher for science mentors but there is a large standard deviation and no statistically significant difference. However, the PhySEP mentor for candidate 14 wrote a very long lesson appraisal and if this candidate were to be removed from the analysis then it becomes statistically different at the 1% level (p=0.01) with science mentors tending to write more.

1

B. Topic specific

Candidate number / PhySEP / Science
% / %
1 / 72.73 / 13.83
2 / 56.86 / 34.78
3 / 64.41 / 15.22 / PhySEP / Science
4 / 80.00 / 30.30 / Percentage Mean / 66.55 / 14.36
5 / 72.00 / 6.00 / Std. Deviation / 13.53 / 12.54
6 / 51.92 / 20.00 / Probability / <0.001
7 / 46.43 / 8.33
8 / 54.90 / 12.50
9 / 55.26 / 1.25
10 / 91.30 / 13.27
11 / 88.24 / 0.00
12 / 78.05 / 8.99
13 / 60.94 / 42.05
14 / 57.33 / 7.69
15 / 67.92 / 1.25

If further analysis is carried out on raw data, we can see a statistically significant difference between the two types of mentor in all three categories of the content analysis.

Because of variations in the number of phrases, all further comparisons are based on the percentage of comments.

For PhySEP mentors part of training focused on making topic specific comments.

C. Control and Discipline

Candidate number / PhySEP / Science
% / %
1 / 0.00 / 26.60
2 / 1.96 / 34.78
3 / 1.69 / 28.26 / PhySEP / Science
4 / 0.00 / 0.00 / Percentage Mean / 2.23 / 38.72
5 / 0.00 / 36.00 / Std. Deviation / 3.87 / 21.68
6 / 1.92 / 63.33 / Probability / <0.001
7 / 7.14 / 83.33
8 / 1.96 / 37.50
9 / 0.00 / 35.00
10 / 0.00 / 14.29
11 / 0.00 / 42.86
12 / 4.88 / 62.92
13 / 0.00 / 13.64
14 / 14.00 / 46.15
15 / 0.00 / 56.25

The difference in the means is highly significantly different, with the science mentors writing more comments about control and discipline. Nearly 40% of the content of science mentors’ written lesson appraisals is focused on control & discipline.

In the training of science mentors, there is encouragement to comment on aspects other than control & discipline but they still tend to focus on this category quite heavily.

D. Generic

Candidate number / PhySEP / Science
% / %
1 / 27.27 / 59.57
2 / 41.18 / 30.43
3 / 33.90 / 56.52 / PhySEP / Science
4 / 20.00 / 69.70 / Percentage Mean / 31.21 / 46.90
5 / 28.00 / 58.00 / Std. Deviation / 12.51 / 18.89
6 / 46.15 / 16.67 / Probability / 0.012
7 / 46.43 / 8.33
8 / 43.14 / 50.00
9 / 44.74 / 63.75
10 / 8.70 / 72.45
11 / 11.76 / 57.14
12 / 17.07 / 28.09
13 / 39.06 / 44.32
14 / 28.67 / 46.15
15 / 32.08 / 42.50

In terms of pedagogy, while the focus of PhySEP mentors is on topic specific pedagogy that of science mentors is more generic.

1

Match between pre-service teachers’ perceptions and reality

For PhySEP mentors

For Science mentors

The PhySEP mentors focus mostly on topic specific matters with minimal focus on generic and control and discipline

Science mentors focus mainly oncontrol & discipline and generic matters with little focus on topic specific

The trends in pre-service teacher perceptions are similar to the actual content of the written lesson appraisals but the degree of emphasis is different.

In broad terms pre-service teachers’ perceptions of the content of written lesson appraisals do match with reality.

1

Future work

Further content analysis of other foci in the appraisals e.g. questioning, advising etc

What patterns are found with mentors from other curriculum areas?

How does the content of written lesson appraisals progress through a placement?

1