Jeremy M. Brown

Jeremy M. Brown

Kathryn Lowe

Jill Fillingham

Philip N. Murphy

Bamforth, M,

Shaw,N.J.

Publication date 2014.

An investigation into the use of Multi Source Feedback (MSF) as a work based assessment tool.

Medical Teacher, 36, 997-1004.

Background

This paper reports on a mixed methods study that investigated the use of Multi Source Feedback (MSF) in one Deanery. Anecdotal evidence from discussions with senior Consultants suggested that there remains some scepticism amongst Educational Supervisors regarding the reliability of MSF as a work based assessment tool. The choice of assessors was the focus of their concerns.

To investigate whether these concerns had any foundation this study aimed to compare Specialist Trainees' hand selected MSF assessor scores with those made by their own Clinical Supervisors. The study also aimed to explore the perceptions of Specialist Trainees and their assessors on MSF as a work based assessment tool.

The intention of this study was to explore the complexities of choice of assessor whilst acknowledging that these variations are ultimately addressed through Norcini’s (2003) three principles that guard against any threats to the reliability of peer assessments: the number of relevant performances observed, the number of peers involved, and the number of aspects of competence being evaluated (p. 541).

The establishment of work-based assessment tools in postgraduate medical education was introduced to add validity and reliability to gauge the performance of doctors in training and ultimately identify doctors who may be in difficulty (Archer et al, 2005).

This has resulted in a significant shift away from the reliance on Clinical and Educational Supervisors’ individual judgements of professional performance. MSF is designed to gather as much feedback as possible to form the basis of constructive discussion between the Educational Supervisor and the trainee. As Wright et al (2013) re-affirm MSF should not be taken in isolation. Moonen-van Loon et al (2013) argue that when making ‘high stake’ judgements a series of work based assessments including mini-Clinical Evaluation Exercise, direct observation of procedural skills, and MSF have to be taken into consideration before a reliable decision on a trainee’s progress can be made.

MSF does though offer a more formalised, systematic, team-focused assessment approach (Archer et al, 2008; Violato et al, 2003). Doctors in training are asked to nominate a number of assessors (at least 12 in this Deanery’s programme) at any one time who then complete the MSF questionnaire confidentially rating the trainees’ performance and fitness to practise in areas such as routine clinical care, team working and communication with patients (Royal College of Psychiatrists, 2012). The trainee also completes a questionnaire as a self-assessment before meeting with their Educational Supervisor to consider the results which are processed centrally and presented as mean scores with verbatim free text comments.

Recent systematic reviews have concluded that the use of MSF to assess both physicians’ and surgeons’ practice is shown to be highly reliable and valid (Donnan et al, 2014; Al Khalifa et al, 2013). There are some questions though that remain about the MSF process. Who gives feedback and how much guidance and training is needed for assessors is explored in the qualitative findings later in this paper. Archer et al (2010) highlighted the risks associated with the unregulated selection of assessors. Cohen et al (2009) reported that there was concern amongst some dermatology trainees about possible victimization by MSF assessors. Bullock et al (2009) also reported discrepancies in assessment ratings between some staff groups and peers (administrators or managers being less likely to raise concerns compared to senior nurses and consultants).

Methods

This mixed methods study consisted of quantitative and qualitative phases of data collection that ran in parallel with each other. Grouped responses of hand selected assessors were measured against those made by Clinical Supervisors. An exploration of the personal accounts of MSF assessors and those being assessed was also undertaken. This allowed the research team to develop understanding of individual situations and perspectives without trying to generalize findings to the wider audience.

Ethical Considerations

This study received University, Strategic Health Authority Health AuthHealth uthorityand NHS Research Ethics and local NHS Trust Research & Development approval.

Confidentiality was assured to all participants. Any identifiable information was removed from interview transcripts. Only members of the research team had access to information. The exception to the maintenance of confidentiality would be solely where unsafe practice was highlighted during interviews. This did not occur.

Quantitative phase

There are different multisource feedback models used but for the purposes of this study the mini-PAT (see figure 1) was chosen (with permission) as it is used extensively in postgraduate medical education (Archer et al, 2008).

Specialist trainees (STs) were emailed asking to respond if they would be interested in taking part in the study. Those that were, were then contacted directly by a member of the study team and after giving their written consent to take part in the study were asked to hand out one mini-PAT questionnaire to a clinical colleague of their choice and the other to their Clinical Supervisor each with an information sheet explaining the study. Each assessor returned their completed mini-PAT questionnaire in a stamped addressed envelope to the research team. Each questionnaire was coded for each ST so the research team could collate the completed questionnaires in pairs. The questionnaires were also colour coded to differentiate Clinical Supervisors (green form) from the hand chosen assessors (yellow form). All data was stored anonymously in SPSS 18.0™ with only code numbers to identify individual participants. Statistical analysis was carried out using the Wilcoxon rank sum test to determine any differences in responses between Clinical Supervisors and hand chosen assessors with respect to the total assessment scores for each of five domains on the mini-PAT (1. good clinical care, 2. maintaining good medical practice, 3. teaching, training appraising and assessing, 4. relationship with patients and 5. working with colleagues) as well as the overall impression of the trainee. The analysis focusses on differences in assessment identified for the sample, rather than for individuals.

Initially, potential recruits to the study were identified by trainee listings supplied by clinical tutors in two large hospitals in the North West of England. A significant number of trainees expressed interest, but despite questionnaires being sent to them, the number of returned pairs of mini-PAT questionnaires was low. In order to recruit more participants, therefore, opportunistic recruitment was carried out by one of the authors (NJS) by attending teaching sessions and asking clinical colleagues in several hospitals in the region to ask trainees to take part.

Qualitative phase

The study population for the qualitative phase was postgraduate doctors in specialist training, hospital Consultants, and nursing staff across one Deanery. Potential recruits were emailed and provided with an information sheet that explained that the project sought opinions and attitudes concerning MSF. The email asked for replies from those interested in taking part in a semi-structured interview that would be digitally recorded. It was undertaken at a time and place convenient to the interviewee. Written consent was taken immediately before the interview started. Semi-structured interviews, which lasted between 20 and 40 minutes, were held with 7 nurses, 7 Consultants and 6 postgraduate doctors. All were experienced in the use of MSF as an assessment tool, either as an assessor or as being assessed.

Analysis of the transcribed semi-structured interviews was undertaken using a thematic framework (Ritchie et al, 2003). All interviews were transcribed verbatim and subjected to further in-depth analysis independently by two researchers (JF and JB) to enhance the credibility of the findings. This phase of the analysis involved both researchers independently identifying key themes before JF coding all interview data into the themes identified. Theme descriptors were defined and re-defined until all data was fully represented (Miles and Huberman, 1994).

Quantitative findings

Forty Specialist trainees took part in the study over a period of eighteen months. The median scores for the responses of the Clinical Supervisors and hand chosen assessors with respect to the total assessment scores for each of the five domains as well as the overall impression of the trainee on the mini-PAT are shown in Table 1. Not all assessors could respond to all questions regarding the trainees (indicating that they had not had the opportunity to observe some behaviours in the workplace) therefore only results for trainees where questions were completed by both assessors were included in the analysis. The profiles of the 40 hand chosen assessors were: 29 Specialist Registrars (SpRs)/STs; 6 nurses; 4 Consultants; 1 Staff and Associate Specialist /Specialty doctor.

Hand chosen assessors’ ratings for good clinical care, maintaining good medical practice, teaching, training appraising and assessing, relationship with patients, and the overall impression of the trainee were significantly higher than those for Clinical Supervisors. Ratings for working with colleagues were the same for both groups of assessors.

28 (70%, n=40) Clinical Supervisors and 28 (70%, n=40) hand chosen assessors made free text comments under ‘anything especially good’. 8 (20%) Clinical Supervisors and 5 (12.5%) hand chosen assessors made free text comments under ‘please describe any behaviour that has raised concerns or should be a particular focus for development’. All 5 of these hand chosen assessors were SpRs/STs. No further analysis was undertaken on the free text comments.

Qualitative findings

Five key themes were identified during thematic framework analysis of the semi-structured interviews. Three related specifically to the issue of selection of assessors: the validity of selecting assessors; anonymity of assessors; and the value of multi professional assessors. Two related to MSF issues more generally: usefulness of feedback; and grading.

Theme 1: the validity of selecting assessors

Self-selection of assessors was a recurrent theme during interviews with STs, nurses and Consultants. Trainees acknowledged they often chose assessors who they had a positive relationship with: You choose people who you know, people you like or like you. (T 4); I think you are always going to pick someone who likes you and you get on well with (T 2). There was an awareness that this could introduce bias into the assessment process: It can be biased can’t it? Someone selecting people who are nice or you know, who they feel are going to give them a nice report (T 2). There was recognition though that constructive criticism was important and it would be damaging if trainees tried to guard against this by avoiding potentially negative assessments:

Being honest, you do select people that you get on with. If I’d had a problem with somebody I wouldn’t give them a form and whether that makes them valid…well it doesn’t make them valid does it because that person’s opinion might be quite important as part of the process. (T5)

Trainees recognised that for this process to be useful they should seek out what they perceived as quite demanding assessors:

I actually quite appreciate constructive feedback and I would tend to ask quite a wide number of Consultants and I will ask people who I know are likely to be relatively strict. (T 3)

Consultants agreed that trainees select people they like and have positive relationships with: It’s only human to select those who’ll give favourable feedback. (C 5); Consultant colleagues as well, they cherry pick. (C 4); Trainees being able to self-select has an impact on the process. (C 1).

Consultants felt that the trainees actively avoided those potential assessors they may have had conflict with: Self-selection, you choose those who you’ve had no conflict with. (C 2); The Speciality Registrars are a bit savvy, they’ve got the opportunity to select out people they don’t get on with. (C 3) It was also felt that they returned to assessors who had given them favourable feedback on previous occasions: They tend to use assessors who have previously given them good feedback (C 1); Obviously they tend to go with people that have given good feedback previously. (C 6)

Some Consultants felt that it was their role to approve assessors: As Educational Supervisors I think they should be asking me; are these appropriate people? (C 4)

Other Consultants felt it would be unfair to be involved in the selection process: Obviously you’ve got to have some method of choosing assessors but it would be unfair to inflict assessors on trainees. (C 5)

Nurses shared the same concerns regarding the validity of the assessor selection process: They cherry-pick of course someone who will say all nice pleasant things about them don’t they? (N 1); It does give them the opportunity for them to select people who might give them a favourable view I think. (N 2); If you’ve had a bad experience with someone and say, oh, no I’m not going to ask them. (N 5) I think it must add bias because of the relationship they would select and choose individuals they think will give them a more favourable response. (N3) They pick particular members of nursing staff who they’ve good relationships with so to some extent it could be manipulated. (N 6)

Theme 2: anonymity of assessors

There were conflicting views around the anonymity of the process. Some trainees felt that anonymity gave transparency to the process: People can say what they really feel (T 5); The anonymous part of it gives people the chance to be honest, ‘cause often people will put their name to things and say that everything’s all right’. (T 1) Some trainees felt that the process was not truly anonymous which had the potential to have a negative impact on feedback: Well you know roughly where it’s come from. You know whether it’s a Consultant or one of the Nurses or one of the SHO’s that’s written it. (T 3);

I’ve actually at the moment got an MSF which is for one of my seniors who erm, I’m not, I find him…he doesn’t really work in a team. I find it difficult when I have to put that, because I may be the only person who he’s asked in the unit, so if I say he doesn’t really help out with the juniors, he’ll know exactly who said that. (T 4)

There were also conflicting views regarding anonymity amongst Consultants. Some felt the electronic version provided appropriate anonymity: The anonymity and speed with the electronic version are more suitable. (C 1) Others felt that even the electronic version lacked anonymity: Because of the way it’s set up it’s not anonymous really. You can tell the comments people make, who it is. (C 3); I think there’s a danger of MSFs not having appropriate anonymity, and I think you have to have some sort of MSF that really, truly keeps people anonymous. (C 4)