Situational Judgement and WMS Curriculum

Contents

1.  Introduction ………………………………………………………………………………………………… 2

2.  Summary and Recommendations ..………………………………………………………….…… 2

3.  Advantages and Disadvantages of SJTs……………………………..…………..………. ………. 3

4.  Recommendations for Present Curriculum ……………………………………………………… 4

5.  Recommendations for New Curriculum……………………………………………………………. 5

6.  Conclusion ……………………………………………………………………………………………………… 7

7.  References ……………………………………………………………………………………………………… 7

8.  Appendix A: the SJT domains………………………………………………………………………….....8

9.  Appendix B: Phase II Student Handbook…..………………………………………………………12

10.  Appendix C: Phase I Situational Judgement material………………………………………..24

1.  Introduction

As of 2013, the SJT has replaced the ‘white-space’ application process as the preferred method of assessment into the UK Foundation programme. This comprises 50% of the total application, with the other 50% from academic achievement.

SJTs are a type of psychological aptitude test also known as ‘professional dilemmas’. Foundation applicants are presented with a variety of hypothetical situations likely to be encountered on the job. Accompanying each situation are multiple ways to respond to the scenario. The test taker is then asked to make judgements about the possible courses of action; either ranking five responses from most appropriate to least appropriate, or by choosing the three most suitable choices from a list of eight options. Answers are then marked according to a pre-determined scoring key.

SJT selection in postgraduate medical training was introduced nationally in 2007 for short-listing candidates at stage 2 of GP recruitment. Here, three ‘non-clinical’ domains: empathy, integrity and coping with pressure, are considered in combination with a clinical problem-solving test. So far, these methods have proved widely successful (Work Psychology Group 2009).

This report looks at what opportunities are presented by the introduction of SJTs for the WMS curriculum – both in its current form and for the revised 2013 curriculum.

2.  Summary of Recommendations

Key points about Situational Judgement Tests (SJT) for curriculum review:

·  SJT domains pin down behaviours outlined by Tomorrow’s Doctors

·  SJTs are a valid way of assessing Non-Technical Skills (NTS)

·  NTS are weighted of equal importance to academic achievement in the Foundation Programme selection process

·  NTS are areas where students can struggle and should be supported as such

·  SJTs are a useful way to reiterate the importance of NTS to students

·  SJTs give an opportunity to evaluate how well we are preparing students for practice

What the SJT can do for WMS in a wider context:

ü  Highlight to the GMC where we are additionally teaching and assessing NTS

ü  Target key areas of assessment within the horizontal teaching themes

ü  Offer a proven way to assess (formatively) students’ knowledge of NTS

ü  Provide material for personal tutors and clinical educational supervisors to actively engage with their students beyond a pastoral role.

ü  Opportunity to promote to prospective applicants how effectively we prepare students for practice.

Recommendations:

Ø  Use the SJTs as a mechanism for evaluating key learning points on professionalism, for example managing own health and challenging prior knowledge

Ø  Use the SJT to draw out professional values from the perspective of the medical student, potentially introducing a system for recording and reflecting on mistakes

Ø  Use case-based learning to consider prioritisation in more detail; for example, listing investigations in order of urgency

Ø  Consider actors for scenarios beyond patient relationships: angry relative, distressed colleague

Ø  Encourage students to communicate and work together as a multi-disciplinary team through scripted group-based scenarios

Ø  Utilise personal tutors to encourage discussions concerning NTS

Ø  Concentrate a few sessions on explaining diagnosis and management to relatives

Ø  Ask FY1’s to feed into the case work to recall what they would manage in a particular scenario, who they would call for help and when. This will engage students in understanding the limitations of the FY1 role.

3.  Advantages and Disadvantages of SJTs

The increasing popularity of SJTs are attributable to the wealth of research supporting this selection method. The advantages of which include (Weekly & Ployhart 2005; De Meijer & Born 2009):

ü  Validity akin to cognitive ability testing

ü  Face validity – ie relevant to the job. This is useful for candidates as well as employers to provide a realistic preview of the proposed role.

ü  Incremental validity of selection above cognitive and personality testing

ü  Unique, multidimensional assessment of performance not captured by other methods

ü  Less adverse impact on ethnic minority groups

ü  Easier to score than other types of assessment such as the situational interview and hence, can be implemented into large-scale testing programmes

However, as with any selection process there are drawbacks. In terms of SJT, the disadvantages include:

û  Variety of performance predictors: unclear as to what they actually measure

û  Multidimensional: difficulties developing scenarios to measure one particular construct

û  Assumes past behaviour can predict future response to a similar situation

û  Debate over whether they are ‘fakable’

Designing an effective selection system requires thorough analysis of the knowledge, skills, attitudes and personal qualities needed to be successful in that role. With respect to foundation selection, the ISFP (Improving Selection to Foundation Programme) developed nine domains of assessment for UKFPO (United Kingdom Foundation Programme Office) application (Patterson et al 2010):

·  Commitment to Professionalism

·  Learning and Professional Development

·  Working effectively as part of a team

·  Patient Focus

·  Problem solving and decision-making

·  Self-awareness and insight

·  Coping with pressure

·  Organisation and planning

·  Effective communication

These domains are discussed in detail in Appendix A. In order to get around the debate concerning ‘fakability’ of SJT (Nguyen, Biderman & McDaniel 2005), the ISFP have introduced answers to ‘should’ rather than what you ‘would’ do, assuming that intentions are related to knowledge. More implicitly than this, they use the SJT to measure those ‘non-technical skills’ (NTS) expected of a junior doctor.

4.  Recommendations for Present Curriculum

WMS can further develop students’ non-technical skills through situational judgement. Given my involvement in the integration of SJT material into the current curriculum, my experiences as a student and through my participation in the student support committee and curriculum review meetings, the following material has been developed for integration:

Phase II:

-  Student workbook (Appendix B): incorporating reading list, interviews with FY1s to carry out (1 per block), observations to carry out, own scenarios to collect (2 per block) to be discussed with clinical educational supervisor (CES)

-  (Collected examples could then be used to feed in to the early years.)

-  4 teaching sessions to be delivered by CES: 1 per block in senior rotation

-  Students to work through online SJT examples in Acute, Obs and Gynae, Child Health and Medicine blocks.

Phase I:

-  Reworked ‘Becoming a Professional’ section of clinical skills 2 handbook to incorporate SJT domains and scenarios. These scenarios are all challenges a medical student may encounter on the wards and loosely cover each domain.

-  Scenarios to be discussed with their personal tutor group before the answers go up online (Appendix C).

5.  Recommendations for New Curriculum

Ø  Commitment to Professionalism: Although professionalism is a horizontal theme in which values and ethics features heavily, there are other aspects less well-approached. Owning up to mistakes is something that medical students should be doing. I think there should be a system for recording and reflecting when students make mistakes as this is what you have to do in practice. Also, responsibility for health is something that the personal tutors can draw out. I feel the tutoring system is presently there for ‘struggling’ students when really students should recognise their limitations before they get to that stage. Relating examples such as this within an SJT-style scenario should draw out such learning points as part of professionalism. Finally, learning to challenge knowledge and speak up is definitely something that could be better drawn out, particularly in the early years. The SJT can utilise these issues and get students engaging in the values now rather than values they think they need in four years’ time. I think the message of the ‘professional medical student’ is key to bring out.

Ø  Coping with pressure: I have to be honest and I don’t think this is taught well in the current curriculum. Whilst students manage the pressure of a heavy workload, the pressures of multi-tasking and prioritising required on the wards are very different. There is a real opportunity to work on this through case-based teaching; investigations and management should be put into context. I think this is something that should be brought into the complex, advanced cases in year two. Consider having a list of investigations for different patients and ask student to prioritise them: e.g. you are on a respiratory ward with x,y, and z, how do you proceed? Another source of pressure comes from difficult patients and colleagues. In this respect the SJT can be useful for discussion as well as assessment. I would also suggest WMS consider actors presenting as an angry relative or distressed colleague. This would be a great way of preparing students for life on the wards beyond patient relationships.

Ø  Effective communication: I think communication – particularly with patients – is very well taught at Warwick. Record-keeping is also mentioned, however it would be good for students to practice filling in the forms they need eg death certificate. What I feel could be brought out more is negotiating with colleagues. As mentioned, an actor could be brought in to do this. You should also consider having a ‘script’ with a profile for each part. One example might be a patient complaint. In groups you could have one person playing a difficult colleague, the other a doctor and one the patient. You can therefore practice relaying difficult information from the patient to your colleague diplomatically. Teamwork and communication are integral to your job as a doctor, yet the difficulties associated with this have yet to be properly addressed.

Ø  Organisation and Planning: prioritisation is something that can be incorporated into coping with pressure. There are also various techniques that students can use to prioritise their workload and perhaps this theory is something that the personal tutors could work on with the students together with an SJT scenario?

Ø  Patient focus: I feel this is the domain that is best approached at Warwick. What perhaps is not as clear however is the amount of focus on patient’s relatives. From interviews on the wards, it is clear that most complaints come from relatives, as do breaches of confidentiality. I think it would be important to concentrate some sessions around explaining diagnosis and management to relatives – either because the patient cannot consent, or because they have asked you to explain it. Again these issues can be drawn out using SJT examples.

Ø  Problem-solving: This is about taking initiative and seeking help. What I feel could be better at Warwick is drawing out what you would deal with yourself as an FY1 and what you would realistically call for help with. It would be nice to have a greater sense of your limitations before you start the role. This could easily be drawn out in case-based discussion alongside clinical knowledge. Knowing the expectations of an FY1 in each scenario would be useful. You could quite feasibly ask a group of FY1’s to feed into your case work and have them say what they would and wouldn’t do for the steps in management and when they would call for help.

Ø  Self-awareness and Insight: Reflection has already been mentioned to be included in the eportfolio. Beyond this, ‘boundaries and limitations’ are part of self-awareness and this is more difficult to teach. Nevertheless, understanding roughly where the limits of an FY1 lie will give students valuable insight into the role they will be doing. This also builds into teamwork and knowing your roles and responsibilities. Bringing in this information earlier will mean that students look out for it more on the wards because they have greater awareness of the NTS they need as well as the clinical skills.

Ø  Teamwork: I have mentioned this in communication but I really think there could be more emphasis on teamwork in the curriculum. The personal tutors can really be utilised in this respect, drawing upon discussions about difficult or struggling colleagues. See Appendix C as examples of this from the current phase I material.

6.  Conclusion

SJT are a fantastic and fun way to get students engaged in NTS. We should however, in all cases, make these skills EXPLICIT to students. Students like to know the fundamentals behind their learning in juxtaposition to the expectations of an FY1. Ultimately it’s about preparing medical students for a job and the SJT really adds value in that respect.

7.  References

·  De Meijer, L.A.L. & Born, M. Ph (2009). The situational judgement test: Advantages and disadvantages.In M. Born, C.D. Foxcroft & R. Butter (Eds.),Online Readings in Testing and Assessment,International Test Commission available from:http://www.intestcom.org/Publications/ORTA.php

·  Ngyen, N.T. Biderman, M.D. McDaniel, M.A. “Effects of Response Instructions on Faking a Situational Judgement Test” Int. J. of Selection and Assessment. Vol 13 (4): 250-260

·  Patterson, F. et al (2010) Improving Selection to Foundation Programme Final Report Appendix D: FY1 Job Analysis. Work Psychology Group and University of Cambridge; pp 126-240

·  Weekly, JA and Ployhart, R.E. (2005) Situational judgment tests: theory, measurement and application. Lawrence Erlbaum Assoc Inc, New York.

·  Work Psychology Group (2009) Evaluation of situational judgement tests to select postgraduate trainees: validation studies in two specialties [pdf] available from: http://www.workpsychologygroup.com/assets/Library/Library-SJTs-to-select-postgrauate-trainees-AMEE.pdf

Frances Varian 19/04/2012

Appendix A

The domain descriptors: definition of the attributes and behaviours expected of an FY1.