Embedding Peer Review of Teaching Into Departmental Practice

1

Embedding Peer Review of Teaching into Departmental Practice.

Clare Kell

Department of Physiotherapy

CardiffUniversity

Paper presented at the British Educational Research Association Annual Conference, University of Glamorgan, 14-17 September 2005

Email:

Tel: 029 2074 2267

Address for correspondence:

Clare Kell

Department of Physiotherapy

CardiffUniversity

Ty Dewi Sant

HeathPark

Cardiff

CF14 4XN

UK

BACKGROUND AND PURPOSE

In response to calls from the Dearing Report (NCIHE, 1999), Quality Assurance Agency and former ILTHE, Higher Education Institutions (HEIs) have been keen to demonstrate the effectiveness of student learning experiences. Focus on the student learning experience has placed emphasis on the teaching environment within which the learning is facilitated. Teaching staff within HEIs have been given the opportunity to internally monitor the students’ learning experiences provided that reports are used to inform any self-reflection document required by auditing bodies (QAA, 2000). Many HEIs operate the dual systems of student evaluation and feedback on their learning experiences, and colleague / peer review or observation of individual learning events or experiences to support their reflective processes. These latter peer-led sessions have formed the basis of much discussion both in the literature and in local academic departments, with rationale ranging from audit / judgmental purposes (QAA, 1996; Shortland 2004), to those focussed on dialogue for mutual benefit and enhancement (Martin and Double, 1998; Orland-Barak, 2005).

The Department of Physiotherapy, CardiffUniversity (formally the University of Wales College of Medicine) piloted a form of Peer Review of Teaching (PRT) in 1999, based in part on the work of Marsh (1987). This first approach to PRT required staff to be paired (non-reciprocally) within their subject teaching teams. No pre-observation meeting took place and the ethos of the observation was a ‘spot check’ of teaching performance. Post observation dialogue was intended to be self-reflective for both parties and was supported by session-specific feedback from the students. Within 2 years the PRT process had run its course; departmental staff were engaging in a wide-range of informal teaching and learning discussions and felt that there was no longer any place for an audit-based process.

During 2001 the Department piloted a peer triad approach (Gosling, 2000). The Peer Triad process enabled small groups of staff to be drawn from across the subject curriculum so that session content became less important. Each Triad was encouraged to select an annual theme and to use the PRT process to both ‘unpack’ each other’s teaching rationale (MacKinnon, 2001), and to engage with specific pedagogic innovations. Triads met as frequently as they wished and each was asked to provide an anonymous summary of observations, thoughts for wider dissemination and possible development needs. Formative PRT proforma continued to help frame the observations: these documents were confidential to the Triad. Feedback suggested that the Triad approach was beneficial to individual members of staff, helped staff explore the published evidence for their innovation and empowered staff to value their ideas and discussions.

In response to NHS workforce planning directives, undergraduate physiotherapy intake numbers more than doubled between 2001 and 2003 with staff recruitment to match; many of the new academic staff were contracted part time to retain NHS service. The Departmental Quality Assurance report of 2003/04 recorded a 67% completion rate for PRT. This low return was unexpected and unprecedented. In response, the Department embarked on a detailed overview of both the ‘new’ Departmental context and the benefits or otherwise of the existing PRT process. This paper reports the qualitative processes that mapped perception of PRT and resulted in the development of a new approach better suited to the changing culture and staff within the department.

METHODOLOGY

All staff were invited to a PRT workshop in protected staff-development time. Staff selected pseudonyms and used these to anonymously, but traceably, respond to the following data collection activities.

Perception of terms:

Blackwell et al (2001) and Gosling (2002) recommend that departments help staff verbalise their perceptions about a PRT process in order to make overt and then clarify issues of urban myth and reality. Using pseudonyms to track responses, participants were asked to reflect on their personal interpretation of the terms: peer, review, teaching and write definitions for each. From these 3 word-lists staff were asked to select one combined definition that best described their perception of what PRT meant for them. Individuals were then randomly paired and asked to discuss their individual definitions before compiling a consensus PRT definition. This final definition carried the pseudonyms of both staff so that the researchers could consider how far each participant had moved from their initial perception in order to generate the consensus.

Reflection about the existing PRT process:

In a paper discussing perceptions of PRT processes, Gosling (2002) offered a table of keywords / phrases to compare three different PRT models (evaluation model, development model, peer review model). Keywords are presented to reflect how each model addresses purpose, outcome, confidentiality issues etc. Each participant was issued with 2 differently coloured pens and asked to 1) circle keywords on the template that best reflected their perception of the existing departmental PRT process and 2) to use another colour to circle those keywords that would reflect their ‘Utopian ideal’ for a PRT process. Responses were again identified using the pseudonym.

Group reflection and discussion:

The activities above were intended to help individual staff reflect on their perceptions of the value and purpose of PRT processes and to consider their personal roles within the existing departmental system. The final activity was an opportunity for staff to verbalise their concerns and discuss in the safety of others. Groups of 4 self-selected a theme for the discussion and used a storytelling, action-learning approach to help one group member explore relevant personal experiences in detail. Each group, again using their pseudonyms, produced a list of emergent key issues.

Data processing:

All handwritten transcripts were sealed and passed to a third party for transcription to ensure that anonymity was preserved.

Data analysis:

A panel of four academic staff reviewed the transcribed data. Independent reviewers generated themes, which through discussion within the panel, led to the creation of templates for each of the ‘definition’ activities. The Gosling (2002) activity was analysed using keyword response frequencies.

RESULTS

78% (n=20) of the Departmental academic staff participated in the workshop. All data was usable and traceable.

Perception of terms:

Table 1 presents the responses to the individual definition activity.. The majority of staff viewed PRT as a reflective process between colleagues that could be used to facilitate the development of the students’ learning experience. 10 of the 44 statements considering the term ‘review’ included elements of a judgmental, summative and audit-like process operated within a ‘power’ dimension.

Key word

/

Definitions: Emergent themes

(examples of exact descriptors in brackets) / Total number of statements per theme
Peer….. / Colleague
(colleague, fellow)
Power match / mismatch
(equal, supportive, senior-led) / 18
18
Teaching…. / Methods
(techniques, giving info, facilitation methods)
Purpose
( Assisting learning, personal development) / 25
34
Review….. / Purpose:
a)Formative
b)Summative
How?
a)Tools (audit)
b)Interaction (discuss)
c)Personal (reflect)
d)Affective / 4
9
10
8
23
3
PRT….. / Methods:
(revisit, qu and evaluation, assessment)
Purpose:
(teaching community, learning development)
Affective:
(positive support, respect) / 21
6
6

Table 1: Response frequencies to themes emerging from individual ‘definitions;’ activity.

Responses from the paired discussion of terms generated similar themes for each term to those presented in Table 1. The notable exception was the disappearance of all words related to audit and summative assessment.

Perception of existing process v utopian ideal.

Table 2 presents the statements from the Gosling (2002) table that were circled. There were a wide range of descriptors used to describe perceptions about the existing departmental PRT process; some staff considered the existing process to be summative and judgemental in nature. Descriptors for a ‘Utopian’ Ideal were more focussed and reflected a formative process being used to promote a culture of learning and teaching discussion and development.

Group discussions.

Staff formed 5 small groups for the discussion activity. All groups discussed issues related to PRT methods and the affective elements perceived to be involved. Written feedback included the following points:

1)All groups thought the process title should be changed

2)Any PRT process resulting from the review day should be time efficient and time protected

3)The process should be supported at all levels with adequate and appropriate training needs met

4)Observation / discussion should occur across the range of teaching contexts and teaching-role activities

5)A need to help staff develop skills in personal reflective practice and reflective dialogue.

6)A unanimous need to overtly acknowledge and address the affective components of PRT

7)Overwhelming support for an empowering, meaningful process that could be streamlined into CPD and other developmental and enhancement activities.

Results summary:

The data collection process, while innovative, was robust and produced responses that were considered to be a true representation of the views of the departmental staff. The results identified a lack of consensus / clarity in terms of rationale, purpose and intended ethos of the existing PRT process. There did appear to be consensus about what a PRT process could achieve. Affective and value statements were a common feature throughout the results.

Department of PhysiotherapyBERA Conference Sept 2005 BERA conference Sept 2005

CardiffUniversity

1

Table 2: Perceptions of existing process v utopian ideal

Statements listed were circled by 10 or more (max 20) members of staff.

* Indicates statements circled by 15 – 17 staff,

** indicates statements circled by 18 – 20 staff

Existing process /

Utopian ideal

How? / Teachers observe each other * / Teachers observe each other *
Purpose? / Identify under-performance, QA, assessment, appraisal
Demonstrate competency
Engagement in discussion about teaching / self and mutual reflection / Demonstrate competency
Engagement in discussion about teaching / self and mutual reflection **
Outcome? / Report / plan
Discussion, wider experience of teaching methods * / Discussion, wider experience of teaching methods **
Status of evidence? / Peer shared perceptions / Peer shared perceptions *
Relationship? / Equality / mutuality / Equality / mutuality **
Confidentiality? / Between observed and examiner
Shared within learning set / Shared within learning set *
Inclusion? / Includes all staff / Includes all staff **
Judgment? / How to improve
Non-judgmental, constructive feedback / Non-judgmental, constructive feedback **
What is observed? / Teaching performance, class materials / Teaching performance, class materials *
Beneficiaries? / Mutual benefit between peers * / Mutual benefit between peers *
Benefits institution *
Conditions for success? / Teacher is valued / Teacher is valued **
Risks? / Risk of complacency, conservatism

Department of PhysiotherapyBERA conference Sept 2005

CardiffUniversity

1

DISCUSSION

Having asked for staff to engage in an honest, open yet time-consuming data collection process, the researchers felt an ethical obligation to involve colleagues in the development of their next PRT process. The following sections therefore, present some of the issues that were discussed with the academic staff and management team.

Ownership.

Giving staff ownership of the PRT process was seen as recognition of the autonomy with which academic staff carry out many of their roles (Shortland, 2004) and likely to result in the development of a flexible process that could be tailored to individual / local contexts and needs (Blackwell and Preece, 2001).

For example, Shortland (2004) deconstructed a university’s PRT documentation and combined the results with direct observation of a PRT triad. The observations occurred over a 2-year period either side of a QAA Institutional Review. Her findings suggested that, when ownership for the PRT process lay with the triad (post- QAA review), staff were empowered to adapt the central policy to suit their own needs (while continuing to meet management’s request for form-filling p. 227). The triad staff recognised the benefits, to themselves as individuals and members of a discussion group, of engaging in a PRT process – the staff shared a ‘new agenda of professional development’ (p. 227).

Rationale and purpose.

The data collected for this study suggested that there was a mixed perception among the staff about the PRT process’ purpose and rationale: established staff saw the process as formative and useful for personal, professional development; newer staff viewed the process as summative and audit-like. While all academic staff within the department of Physiotherapy, CU are state registered physiotherapists, recent changes within the NHS sector have increased focus on competency to practice. As such, practising therapists are regularly reviewed by their line-managers. To newer academic staff any process involving the observation of practice was thought therefore to be summative, audit-like and seniority- / management- led. While efforts had been made to integrate newer staff into the departmental PRT process, established staff were unaware of this perceptual difference. The discussions that followed presentation of the study results helped all staff to discuss and clarify issues of process rationale and purpose.

Both management and academic staff wanted a process that would help assure the quality and efficiency of student learning experiences (Clegg et al, 2002). All parties wanted focus to be taken away from content / stand-up performance and instead embrace the breadth of the ‘teaching’ role and its impact on the total learning environment (Radloff, 1998; Hammersley-Fletcher and Orsmond, 2005). Academic staff (established and new) agreed that PRT, and the ensuing reflective dialogue between peers, to be beneficial both to their personal professional development and teaching development across the whole department (Shulman, 2002).

Ethos:

The study results exposed a lack of clarity about the ethos of the existing PRT process. While descriptions of an ‘Utopian’ Ideal were fairly consistent, the existing process was viewed as formative by some and summative by others. Informal discussions with staff who had not engaged with the PRT process in 2003/04 generated two main themes: lack of time, and a previous review that was so judgmental and ‘cloning’ that they had refused to take part again – even though triad groups had changed. Keeping identities confidential, this latter problem was discussed with the staff. All staff worried about having a similar experience.

Many authors advise that reflective practice for the mutual benefit of both the reviewer and reviewed is an essential component of any PRT process designed to enhance and value learning and teaching diversity (Hammersley-Fletcher and Orsmond, 2004; Beaty, 1998). Reflective practice based dialogue, occurring in an environment of trust and mutual ownership, will help both parties ‘unpack’ their practises as instinctive teachers (MacKinnon, 2001) and go beyond the assumptions both take for granted in their approaches to teaching (Pill, 2005; Paris and Gespass, 2001).

All state registered physiotherapists are required to undertake Continuing Professional development (CPD) activities to maintain their registered status. Reflective practice is a core ethos of these activities. While staff had experience supporting the development of reflective practice skills in their undergraduate students, many were concerned about the skills required to facilitate peer reflection (Hammersley-Fletcher and Orsmond, 2005). The researchers were requested to append examples of reflective practice models, sample questions and key ‘giving formative feedback’ points to any new process paperwork. Management also agreed to support a staff development afternoon considering issues related to the giving and receiving of verbal feedback.

Affective issues:

The data collection discussions had raised staff concerns about affective issues related to any PRT process. Even within a process that focussed on mutual development, there was concern for ‘pulling of rank’, ‘undercurrents of power games’ etc. Overt discussion of these concerns led staff to agree that the term ‘Peer’ referred to an equal with respect to learning and teaching activity i.e. although one member of staff may be more administratively senior to their triad, all were equal in the functions of teaching that they performed (Gosling, 2003). The term ‘Review’ was felt to be unhelpful with its judgmental undertones for practising clinical staff. Staff selected ‘Reflection’ to describe the new process acknowledging the trust (Allen, 2002) and reciprocal collaboration (Martin and Double, 1998) at its foundation.

Data collection:

Shortland (2004) describes the common mismatch between rhetoric and practice when staff are asked to evidence the observations they conduct e.g. formative rhetoric supported by closed question, ranking proforma. Staff discussed some common options (Shortland, 2003; Gosling, 2002; MacKinnon, 2001), the quality of data each was likely to produce and the influence that this data might play on the quality of the reflective practice that was possible (Pill, 2005). It was agreed that the new process would encourage staff to use a variety of data /evidence collecting options to best suit the needs of the individuals and potential triad innovations. Data collection transcripts would continue to be confidential to the parties directly involved with central returns being required for completion data only. Triads are asked to feed interesting practice issues and possible suggestions for role support and development into the programme review processes. These returns could be anonymous.