Evaluations – what lessons can we learn from these?
Paper presented at the British Educational Research Association Annual Conference, University of Glamorgan, 14-17 September 2005
Richard Harris
School of Education, University of Southampton, UK
Background and aims
There is a debate about how trainee teachers develop across the course of the training year. Fuller and Bown’s model (1975) based around three broad stages of development, from a focus on self/survival, to task/mastery of teaching situation and then finally on pupils, has been influential. Kagan’s (1992) review of studies offers support to this stage theory model, and though Conway and Clark (2003) offer a modified version of this model, they claim “the outward trend in stages of concern posited by Fuller was manifested by the prospective teachers in this study” (p.470). Though Furlong and Maynard (1995) offer a more complex outline of trainee teachers’ development, which they see as broad patterns rather than linear stages, their five stages do appear to follow Fuller and Bown’s original model.
This model has been challenged. Pendry (1997) in her study of trainees’ thinking at the lesson planning stage claims “there was no evidence in this study of common stages of development for the interns … there was no evidence of these beginning teachers showing an initial preoccupation in their planning with management and survival concerns” (p. 95). Burn et al (2000, 2003) conclude “the attempt to reduce the process of learning to teach as a series of discrete stages obscures not only the complexity of that process, but also the enormous variation between individuals in terms of their starting points and the ways in which their thinking develops” (2003, p. 329). These findings support the earlier claims of Guillaume and Rudney (1993) that trainee teachers hold simultaneously a range of concerns that continue over the course of the training year, though the nature of the concerns may change.
These differences may be due to context or the focus of the studies. Those that support the notion of stage theory focus on classroom performance whilst the work of Pendry and Burn et al has as its prime focus decision making prior to teaching. In addition, Conway and Clark (2003) attribute the difference to studies that are either looking for variety of concerns as opposed to the most salient of concerns. This suggests that we need to find some way to reconcile these differences. Essentially there are four questions to answer:
- what issues concern trainees during their development as manifested by their written evaluations (both positive and negative comments)
- what are their main concerns?
- do the nature of their concerns alter during the course of their training?
- is it possible to discern any pattern in their concerns during their training?
Methodology
Loughran (1996) used reflective journals as a means of gaining an insight into trainee teachers thinking and concerns. Macintosh (1998) though questions the suitability of such journals or diaries for gaining such an insight, raising ethical questions about the use of reflective journals and their practical value, calling them “time-consuming, repetitive and of largely superficial descriptive content” (p. 555). This study therefore focused on lesson evaluations as offering a potential insight into trainees’ concerns. Lesson evaluations occur as a natural part of the training course, have a clear purpose in helping trainees reflect upon the strengths and weaknesses of their teaching and feed into target setting for the trainees. As such it is assumed that the evaluations would reflect the immediate concerns of trainees as they develop through the course. The evaluations should show whether trainees comment upon different things at different stages during the year, and therefore whether trainees develop through stages. In addition it was expected that the evaluations would offer an insight into both the planning process and the actual classroom performance. Combined with follow up interviews in pairs, it should be possible to analyse the findings from the evaluations to look for change and continuity in concerns and determine which, if any, issues are seen as being of most concern. By using evaluations that span the entire year, rather than observations and interviews at discrete points, a more detailed picture of thoughts and concerns would be discernable. The use of evaluations should allow access to immediate reflection, whereas the interviews would allow for more considered thought across the year and so providing a form of triangulation. Though the study is small scale, previous studies such as Conway and Clark’s (2003) were of a similar size and have yielded rich and valuable insights into our understanding of beginning teachers’ development, and as Hammersley (2001) shows will add to the ‘mosaic’ of our understanding.
Sample
The data for this study come from a 1-year History Post-Graduate Certificate in Education (PGCE) course. Five trainees took part in the study (initially there were six, but one did not provide evaluations for all stages of the course).
Data collection
In total 262 lesson evaluations were offered for analysis (see table 1).
Table 1 Number of evaluations handed for analysis by trainees at different stages in the year
Trainee A / Trainee B / Trainee C / Trainee D / Trainee E / TotalPre-Christmas / 12 / 8 / 23 / 24 / 25 / 92
Pre-Easter / 16 / 11 / 12 / 16 / 24 / 79
Post-Easter / 9 / 24 / 16 / 17 / 25 / 91
Total / 37 / 43 / 51 / 57 / 74
The evaluations were collected from three phases of teaching. A five week teaching placement prior to Christmas in one school and a later 12 week placement in a second school. This second placement involves six weeks teaching prior to Easter and six weeks after Easter. Evaluations were therefore collected from the trainees for these three periods of teaching.
In their evaluations trainees identify three things that they felt went well in the lesson and three things that could have been improved and an open-ended comment. Before Christmas trainees had to evaluate all lessons, whilst during the second school placement they had to do a daily evaluation.
Interviews were held at the end of the course. Interview questions arose from analysis of the evaluations and were focused primarily on the trainees’ concerns and the extent to which these altered across the year. Trainees were asked about the value of evaluating their lessons and the perceived benefits and shortcomings of the model they used. The trainees were interviewed in pairs (this included all 6 trainees who initially started the project), based on friendships; interviewing people together is seen as useful “where a group of people have been working together for some time or common purpose” (Watts and Ebbutt, 1987, cited in Cohen et al,2000, p.287) and the use of friendship pairings would allow the participants the opportunity to be more open in their discussions. Prior to the interview trainees were allowed to look back over their evaluations as stimulus and during the interview, bar graphs showing the number of comments recorded under each category were shared for discussion.
Data analysis
The coding stage appears crucial in understanding how trainees develop. Haritos (2004), using the same codes as Fuller and Bown (1975) of survival, teaching situation and pupil, found a different outcome; however this is due to what was included within each category. For example, Fuller and Bown include classroom management issues under survival, whereas Haritos includes these under pupil.
It was therefore important that the evaluations were coded inductively in a process similar to that described by Altrichter et al, 1993. Following this initial stage, the process became iterative as new material was collected and literature consulted. Initial coded comments from the data were regrouped into five main categories: teacher actions (TA), pupil response (PR), pupil learning (PL), tasks/resources (TR) and self (S). These categories bear a close resemblance to those identified by Burn et al (2000, 2003). A second coder analysed a sample of evaluations and created their own categories independently. Discussion led to agreement over the five categories that were finally adopted.
Over 1600 units of analysis were coded and categorised. To judge whether the nature of concerns altered across the year, two of the categories (pupil response and tasks/resources) were analysed and sub-categories created.
Analysis of the interviews was based around key questions. These had emerged from the earlier analysis of the evaluations and an initial reading of the transcripts to get a feel for what had been said. These were based around:
- What were trainees concerned about (focusing on main concerns at different stages in the year)?
- Did the nature of concerns change across the year?
- Did the trainees feel they had developed through discrete stages?
- Did the trainees feel the graphs showing the categories reflect how they developed across the year?
- What factors influenced what was included in the evaluations?
Findings
An analysis of the evaluations shows a very mixed picture about trainees’ concerns across the year. Taken as a whole (see figure 1), the comments about teacher actions are the largest at all stages of the year, but do decline from over 40% of all comments to 30%. Comments about pupil learning and task/resources increase steadily during the year, whereas comments about pupil response and self fluctuate.
Figure 1
The broad categories conceal a range of issues. Teacher actions covers the widest range of concerns, covering 15 sub-categories; from minor concerns about knowing pupils’ names and use of voice to a large number of comments about explaining learning objectives, concepts, tasks and so forth. Pupil response comments include the degree of enjoyment/engagement from the pupils, pupil behaviour and noise levels from pupils. Pupil learning includes comments about knowledge, skills or understanding that pupils demonstrated, or the amount of work that was completed. Task/resources is about both the type of task pupils tackle and the suitability of the task or resources for the pupils. Self is to do with self-confidence, which manifests itself through comments about relationships with the classes or individuals, or about subject knowledge matters. Four trainees had concerns in all five categories throughout the year. Only one trainee (trainee D) during their first phase of teaching did not comment at all about pupil learning or self. In the teaching prior to Easter this same trainee still had no comments about self, though this had changed by the final phase of teaching.
Individual profiles of trainees, however, do not match the overall picture shown in figure 1. Table 2 provides a comparison of the percentage of comments relating to the different categories across the year. All the trainees show a decrease in comments about teacher actions, apart from trainee C. The starting points for each though are very different. Trainee B’s comments about teacher actions initially account for two thirds of all comments and remain high across the year. For trainees A, D and E, teacher action is no longer the most commented upon aspect of teaching by the end of the course. Comments about self start from a low base and tend to remain low across the whole course, except for trainee C, who shows a sharp rise between the first two phases of teaching.
Table 2Percentage of comments for each trainee by main categories at different points in the training year
TA (%) / PR (%) / PL (%) / TR (%) / S (%)Trainee A
Pre-Christmas / 34 / 23 / 17 / 14 / 11
Pre-Easter / 33 / 27 / 23 / 16 / 1
Post-Easter / 19 / 28 / 30 / 21 / 2
Trainee B
Pre-Christmas / 67 / 19 / 0 / 15 / 0
Pre-Easter / 48 / 14 / 8 / 30 / 0
Post-Easter / 37 / 14 / 17 / 30 / 1
Trainee C
Pre-Christmas / 47 / 21 / 5 / 20 / 7
Pre-Easter / 53 / 12 / 3 / 12 / 20
Post-Easter / 43 / 20 / 7 / 18 / 12
Trainee D
Pre-Christmas / 34 / 32 / 13 / 18 / 4
Pre-Easter / 26 / 23 / 11 / 23 / 8
Post-Easter / 24 / 37 / 11 / 20 / 8
Trainee E
Pre-Christmas / 38 / 21 / 12 / 28 / 1
Pre-Easter / 32 / 14 / 24 / 26 / 4
Post-Easter / 23 / 26 / 22 / 28 / 1
The evaluations also reveal trainees see a complex interaction of elements that influences their ability to teach well. For example during the pre-Christmas placement, trainee C wrote:
I have serious issues with behaviour management. Pupils whining, calling out my name and messing about. A nightmare lesson, completely my fault – the task was pitched too high for them, and behaviour problems meant that I couldn’t explain the task very well. I’ve also been told that I can’t rely on PowerPoint.
Here we can see that pupil behaviour and the task are closely linked. In this situation the teacher’s actions become more problematic. Such connections do appear in several evaluations though their frequency is not high. It seems the style of evaluation does not seem to encourage trainees to express these links. However it is clear during the course, whilst carrying out observation of lessons and post-lesson discussions that the trainees are making these links.
Fuller and Bown’s model (1975) suggests concerns with self predominate early on to be replaced by other concerns. These evaluations suggest that concerns with one’s own actions as teacher (which fit more comfortably into Fuller and Bown’s notion of mastery of teaching situation) are a high priority to start with, which can become more acute for some trainees. The move towards task and then pupils also seems to be more complicated than the model suggested by Fuller and Bown. The number of comments about task/resources, pupil response and pupil learning fluctuate and show no discernable trend across the year. It is interesting to note which categories are most commented upon at the start and end of the year. Trainee A moves from teacher actions to pupil learning, trainees B and C continue to focus on teacher actions, trainee D moves from teacher actions to pupil response, whilst trainee E shifts from teacher actions to task/resources.
Data from the interviews presents a slightly different view. Initial concerns were centred on behaviour. This concern was short lived and was replaced by lesson timings, giving instructions and explaining lesson objectives. Once trainees moved into their second school they talked about a wider range of issues. Settling into a new context raised concerns about behaviour for two trainees, but for all of them they concentrated more on their general teaching skills and the thinking that went into their planning. For example trainee A talked about “I’m concerned about discipline and I’m concerned especially about a couple of the new classes that I’ve taken on”, but also:
I think the things I was worried about at this stage was teaching and learning styles whereas before I was just worried about planning the lesson, getting through the lesson, the beginning and the end, now because I’ve got that sorted pretty much, it seems to be, I’m more worried about the teaching itself ‘am I using visuals enough?’ ‘Is it kinaesthetic?’ ‘Is there ICT in it?’ ‘is there stuff for all the different learners?’ which just shows that you progress away from the mechanics of the lesson more to teaching and learning styles and all the things that are a bit meatier really.
Surprisingly, by the end of their teaching, trainee E, who had previously been little concerned by behavioural issues, found that this was a growing concern even though her teaching was progressing well and the behaviour of her classes was not a problem, she seemed far more sensitive to minor disruptions.
The findings from the interviews seem to reflect generally the stage theory model of progression more closely than the evaluations show. There are initial concerns with behaviour and survival, that progress to a preoccupation with teaching skills before finally focusing on extending the pupils’ understanding of the subject, though it is clear that some concerns persist, reappear or emerge at unexpected points in the course.
Trainees were asked directly whether they felt they had developed according to a stage theory model. Two trainees initially agreed with the stage theory model, but on reflection did not feel it truly represented the thinking that they put into their teaching. As trainee D says:
I don’t think we were just concerned about ourselves at the beginning…I know we said behaviour management and time management but then we both said we quickly moved on to were we making our objectives clear and I started thinking about that in my first placement.
It was felt the model was too generalised and did not reflect the context within which trainees had to operate. According to trainee C “I’m sure it depends on the experiences you have as you first teach your first class as to what you’re concerned about”. This was a point reinforced by trainee D who found her move to her second school difficult: “I struggled more there at the beginning but with behaviour management and that was probably why I was worried about my confidence and things like that and then”.
Bar graphs showing the number of comments under the five different categories used to code evaluations were shown to trainees for comment. The trainees all agreed they had focused a lot on their own actions, particularly towards the start of the year. As one trainee pointed out though, this should not be a surprise as the focus of the course and lesson observations is on them as teachers and how they develop: