A descriptive analysis of the use of workplace-based assessments in UK surgical training
Joseph Shalhoub1, Cristel Santos2, Maria Bussey2, Ian Eardley2, William Allum2
1 Department of Surgery & Cancer, Imperial College London, UK
2 Joint Committee on Surgical Training, UK
For Correspondence
Mr William Allum
ISCP Surgical Director
Joint Committee on Surgical Training
35 - 43 Lincoln’s Inn Fields
London WC2A 3PE, UK
E-mail:
Tel:+44 (0) 20 8661 3982
Word Count
3,266 Words
Abstract
Background
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007. The study aim was to describe use of WBAs by UK surgical trainees and examine variations by training region, specialty or level of training.
Methods
The database of the Intercollegiate Surgical Curriculum Programme (ISCP)was interrogated for WBAs between August 2007 and July 2013, with in-depth analysis of two periods: August 2011 to July 2012, and August 2012 to July 2013.
Results
Numbers of validated WBAs per trainee per year increased more than 7-fold, from median 6 per trainee in 2007-2008, to 39 in 2011-2012, and 44 in 2012-2013.In 2011-2012, 58.4% of core trainees completed the recommended 40 WBAs; only 38.1% of specialty trainees achieving 40 validated WBAs. In 2012-2013, these proportions increased to 67.7% and 57.0% for core and specialty trainees, respectively. Core trainees completed more WBAs per year than specialty trainees in the same training region. London core trainees completed the highest numbers of WBAs in both 2011-2012 (median 67) and 2012-2013 (median 74). There was a peak in WBAs completed by London specialty trainees in 2012-2013 (median 63). The most validated WBAs were completed by ST1/CT1, with a gradual fall in median WBAs to ST4, followed by a plateau; in 2012-2013 there was an increase in WBAs at ST8. Core surgical trainees complete ~50% ‘operative’(PBA/DOPS) and ~50% ‘non-operative’assessments (CBD/CEX). During specialty training, PBAs represented ~46% of WBAs, DOPS 11.2%, CBD ~23% and CEX ~15%.
Conclusions
UK surgical trainees are, on average, undertaking one WBA per week. Variation exists in use of WBAs between training regions. Core trainees tend to use the spectrum of WBAs more frequently than their senior colleagues. Further work is required to examine the role of WBAs in assessment, and engagement and training of trainers in processes and validation of WBAs.
Key Words
Workplace-based assessment, surgical training, Intercollegiate Surgical Curriculum Programme
Introduction
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007 to accompany the competency-based Modernising Medical Careers postgraduate training programme1. WBAs are delivered as part of the Intercollegiate Surgical Curriculum Programme (ISCP), a joint activity by the four surgical Royal Colleges2. The ISCP has provided each surgical specialty with a comprehensive syllabus, a teaching and learning framework, an assessment system, a repository for evidence on individual trainee progress and an interactive web platform.
The use of WBAs is a mandatory component of each surgical trainee’s portfolio through both core surgical training and specialty training. The main WBA assessment tools include procedure-based assessments (PBA), direct observation of procedural skills (DOPS), clinical evaluation exercise (CEX) and case-based discussion (CBD),as well as multi-source feedback (MSF). A glossary of terms used is presented in the Appendix.
PBA involves direct observation of an index procedure or operation with comments given on important steps, tasks or skills at the pre-, intra- and post-operative stages of the procedure, considered to be essential for its safe and successful completion. The assessment form for a PBA is specific to the procedure or operation being assessed. Similar to PBA, DOPS are more generic and relate to either less complex procedures or parts of a larger procedure. The domains of assessment in DOPS are generic and fixed and considered to be transferable across this spectrum of tasks. CEX is an assessment of a trainee’s clinical skills, for example history taking, clinical examination, and information giving. CBD refers to a formalized discussion related to the care of a patient with a focus on knowledge and attitude.Each of these WBAs is completed by both trainee and trainer, validated by the trainer and is recorded within ISCP.MSF is an opportunity for members spread across the multidisciplinary health care and administrative team to offer anonymous feedback on the trainee. The results are then discussed at a meeting with the assigned educational supervisor and the trainee’s annual review of competence progression(ARCP)3.
At present, the Joint Committee on Surgical Training (JCST) recommends that each trainee completes a minimum of 40 WBAs per year of training. This approximately equates to one WBA per week, although trainees and trainers are encouraged to complete more with a spread throughout a training post to demonstrate progression of clinical and technical skills5.
The purposeof this study is to report on the use by surgical trainees in the UK of the largest platform for competence-based training worldwide. Weaim to describe the use of WBAs by surgical trainees and to examine any variations observed by training region, surgical specialty or level of training.
Methods
Every assessment recorded in trainees’ portfolios is recorded with background information about the individual trainee. These data were collated to facilitate analysis to quantify the use of the WBAs. The ISCP database was interrogated for WBAs linked to trainees’ portfolios between August 2007 and July 2013, with an in-depth analysis of two periods: August 2011 to July 2012, and August 2012 to July 2013. These two time periods represent the most recent two years for which, at the time of data analysis, complete data was available.Trainees in Core Surgery and all but one of the surgical specialties were included from 2007. Vascular surgery only became a separate specialty in 2012 but has been included for the second time period. Trauma and Orthopaedics trainees were variably included in the first years of ISCP as they were also able to use the Orthopaedic Competence Assessment Project (OCAP) system; all had transferred to ISCP by 2012/13.
Medical graduates in the UK commence postgraduate clinical training with a 2-year generic ‘Foundation Programme’, from which candidates apply through a competitive nationalselection system for core surgical training (CST) programmes – previously termed the senior house officer (SHO) grade. Following successful completion of CST and the Intercollegiate Membership ofthe Royal College of Surgeons examinations, trainees apply through competitive national selectionfor higher surgical training (registrar grade) in one of the 10 surgical specialities6.
The main WBA assessment tools included in the analysis were DOPS, PBA, CEX and CBD for all trainees in both core and higher specialty training, although there was some data available for assessment of audit, teaching and multi-source feedback. All trainees with an ISCP appointment type of core surgery (CT), fixed-term specialty training appointments (FTSTA), locum appointments for training (LAT), SHO, specialist registrars (SpR), specialty registrars (StR), and StRs converted from SpR were included in the analysis. CT1 and CT2, as well as (for ‘run-through’ specialties/programmes) ST1 and ST2 were combined for analysis for consistency of reporting on the early years of training.
Only WBAs which had been trainer validated were included. Entries with missing variables were excluded. Data were anonymised and allocated a unique identifier to avoid duplicate counting, and analysed by training region (Local Education and Training Board [LETB] in England and Deanery in Wales, Scotland and Northern Ireland), surgical specialty, and level of training at the time of the assessment. Records without appointment type, training region,specialty or training level were excluded from further analysis. After data cleaning, data comparison and summary statistics were performed using STATA version 11 (StataCorp LP, Texas, USA).For the analysis,medians and percentiles were used to represent the data as these are not affected by extreme values.
Results
Records without appointment type,training region,specialty or training level totalled 1,433 for 2011-2012 and 1,355 for 2012-2013; these records were excluded from further analysis. A total of 754,165 ISCP WBAs were validated by UK surgical trainees between August 2007 and July 2013 (Table 1). Approximately two thirds of trainees includedare male (Table 1). There has been an increase in both the number of WBAs validated using the ISCP and the number of surgical trainees using the ISCP for WBAs year on year between 2007 and 2013. The number of validated WBAs per trainee has increased more than 7-fold, from a median of6 per trainee in 2007-2008, to 39 per trainee in 2011-2012, and 44 per trainee in 2012-2013.
In 2011-2012, 55.5% of core surgical trainees had completed the JCST-recommended 40 WBAs, with only 35% of specialty trainees in surgery achieving 40 validated WBAs. In 2012-2013, these proportions increased to 65.5% and 53.6% for core and specialty trainees, respectively Figure 1 illustrates the number of WBAs being completed by core and specialty trainees for these two time periods.
WBAs across training regions
The use of WBAs across training regions for 2011-2012 and 2012-2013 can be seen in Table 2. For the year 2011-2012, the median number of WBAs undertaken by trainees varied from 30 to 50 (Figure 2a). For 2012-2013 this variation was between 39 and 67, with the peak number undertaken by London trainees. After separating core and specialty trainees (Figures 2b and 2c), core trainees tended to complete more WBAs per year than specialty trainees in the same training region. London core surgical trainees completed the highest numbers of WBAs in both 2011-2012 (median 67) and 2012-2013 (median 74). There was also a peak in WBAs completed by London specialty trainees in 2012-2013 (median 63).
WBAs across core surgical training and surgical specialties
The use of WBAs across core surgical training and individual surgical specialties for 2011-2012 and 2012-2013 can be seen in Table 3. The median number of WBAs completed by core surgical trainees increased from 43 in 2011-2012 to 48 in 2012-2013. Excluding vascular surgery, which became a recognised surgical specialty in 2012, the median number of WBAs completed by specialty trainees in 2011-2012ranged from 21 in trauma and orthopaedic surgery, to 41 in neurosurgery, oral and maxillofacial surgery, and otolaryngology. By 2012-2013, this range increased to between 39 in paediatric surgery, and 45 in otolaryngology and vascular surgery. For trauma and orthopaedic surgery, the median number of validated WBAs almost doubled from 21 in 2011-2013 to 41 in 2012-2013 reflecting the change from OCAP to ISCP.
WBAs across surgical training levels
The use of WBAs across surgical training levels for 2011-2012 and 2012-2013 can be seen in Table 4. The most validated WBAs were completed by ST1/CT1 trainees, with a gradual fall in the median number of WBAs to ST4. This is followed by a plateau, however in 2012-2013 there was an increase in WBA numbers at the ST8 training level.
Types of WBAs being used
Analysis of the type of WBA shows core surgical trainees complete approximately 50% ‘operative’or interventional(PBA and DOPS) and 50% ‘non-operative’assessments (CBD and CEX), with the remainder (namely assessment of audit, observation of teaching and multi-source feedback) contributing a small proportion (Figure 3a). During core training, PBAs contributed approximately 14% of validated WBAs, whilst DOPS represented about 31%; CBDs and CEXs were equally split at about a quarter of validated WBAs each. During specialty training, PBAs represented approximately 46% of WBAs, DOPS 11.2%, CBD about 23% and CEX approximately 15%; the remaining WBA types again contributing a small proportion (Figure 3b). The proportion of WBA types remained relatively stable from 2011-2012 to 2012-2013.Variability in the type of WBAs performed across the specialties was observed.
Discussion
The ISCP has offered a curriculum for surgical training and a platform for monitoring of the competency-based training that has accompanied the Modernising Medical Careers era of postgraduate medical training1. At present, competence in this system is being assessed primarily through the use of WBAs. Few would disagree that the introduction of WBAs to support competency-based training is one of the most significant changes to educational policy in medical and surgical training in recent years.
The ISCP is designed to allow the trainee to demonstrate progression in knowledge and clinical and technical skills, as well as achieving professional capabilities using WBAs as formative tools. Engagement with this approach to training has required faculty and trainee reinforcement of its principles and, while acknowledging a specific number should not be an absolute, the figure of 40 has been selected as it approximates to a minimum of one WBA per week. For each training year, there are expected levels of competence defined within the curriculum which are reviewed on an annual basis by a team comprising a programme director, independent educational supervisors, the postgraduate Dean and external specialty advisors. At this review the formative assessments undertaken by clinical supervisors, together with an overview report from an assigned educational supervisor, are considered in a summative way to confirm the appropriateness of a trainee proceeding onto the next year of training.
Whilst WBAs have been a part of surgical training since 2007, from the 2012-2013 academic year the American Board of Surgery has required the completion of in-program assessments 4. These are broadly divided into operative performance assessments and clinical performance assessments. At the time of implementation in 2012-2013, two of each of these assessments was required, increasing to six of each from the 2015-2016 academic year4. The Operative Performance Rating System (OPRS) is employed for operative performance assessment. OPRS WBAs are broadly similar to PBAs in that they are operation-specific, examine and assess defined steps of a procedure, with each step scored against a 5-point Likert (as compared with ‘satisfactory’and ‘needs development’used for PBAs). For clinical performance assessment, similar to CEX in the UK, the American Board of Surgery uses mini-CEX and Clinical Assessment and Management Exam –Outpatient (CAMEO) for the direct observation of clinical assessment of patients.
The purpose of this study was to examine the use of WBAs across a number of denominators in UK surgical training.This is the first time that data describing the use of WBAs and their validation has been released by the ISCP. The data reflects that there has been a good uptake with regards the use of WBAs since the commencement of their use in 2007, with a steady rise in WBAs per trainee per year from 2007 to 2013.
However, the proportions of trainees who are meeting the JCST’s set minimum for validated WBAs of 40 per annum – although rising – remains below 70% and 60% for core and specialty trainees, respectively, for 2012-2013. It may be that these trainees are completing 40 WBAs, but these are not being validated.At present there is a perceived emphasis on the numbers of WBAs completed, as compared with their content; this may be driven, at least in part, by the minimum numbers requirement.While this may show engagement with training it is essential that quantity does not trump quality7.
One of the implications of this study is to highlight that additional support is necessary for trainees to facilitate their use of WBAs, but importantly there is undoubtedly a need to ensure that trainers are engaged, trained and supported by Deaneries / LETBs in the processes related to WBA completion and validation. Mechanisms for this include through accredited courses, for example Training the Trainers (TtT), Training and Assessment in Practice (TAiP) and Training and Assessment in the Clinical Enviroment (TrACE), and through the Faculty of Surgical Trainers (affiliated with the Royal College of Surgeons of Edinburgh).
Core trainees are completing more WBAs than specialty trainees but it is expected that as trainees who have ‘grown up’ with WBAs become the majority within specialty training, this difference will be minimised.There is no clear difference with regards the uptake of WBAs by different specialties, in particular the numbers for trauma and orthopaedic surgery have come up to the level of other surgical specialties in 2012-2013 following the transition onto ISCP from OCAP.
The variation in completed WBAs by training region highlights differences in approach across the UK by Deanery / LETB. For example in London there was a requirement in February 2012 for trainees to complete 80 per annum8. This requirement has resulted in a peak in WBAs in London from both core and specialty trainees in 2012-2013. Despite this peak, the medians remained below 80 per annum at a median of 74 and 63 per annum for core and specialty trainees in surgery, respectively.Powell and colleagues specifically evaluated trainee and trainer opinion with regards increasing the number of WBAs per trainee per year7. Several concerns were highlighted including a reduction in efficacy with increasing number, an increased burden on trainers and the development of trainees focussed on quantitative rather than qualitative outcomes. There is however no clear guidance in the optimum number and further evaluation is required.
The dominance of ‘operative’ or interventional WBAs, particularly PBAs, amongst specialty trainees is perhaps unsurprising. Work examining WBAs undertaken in Wales emphasised that usage amongst specialty trainees is skewed towards PBAs, with the authors concluding that the annual incremental uptake of WBAs between 2007 and 2013 “is reflective of the acceptability and reliability of the PBA and the increasing confidence of trainers and trainees with WBAs”9.However WBAs should also be used to evaluate non-operative skills such that the more equitable proportion of the different WBA tools seen in Core training should be re-established in Specialty training.