A guide to the ES Review for the Lay Assessor
This guide has been prepared ahead of a training session for lay assessors on Sept 1st run by the Yorkshire &Humber Deanery (Y&HD).
Excellent work on Educational Supervision (ES) and the Annual Review of Competence Progression (ARCP) panel process has already been done by Dr Ramesh Mehay who has produced Ram's ARCP in 10 steps (also known as ARCP checklist – detailed) and Dr Paul Johnson who has prepared an Educational Supervision Meeting Trainee Preparation Checklist (Click here to download).
You may also find a wealth of information on the WPBA components of the MRCGP at
The aim of this document is to try and describe the meaning of the different parts of the Educational Supervision Review (ESR) Process; hopefully avoiding too much by way of duplication of the above guides.
Disclaimer Alert! They represent my understanding of the process as a GP Tutor, ARCP panel chair, Educational Supervisor and Trainer. There are likely to be other views but they should be in line with the views of the Deanery Assessment Group (DAAG) of which I am a member. I would like to thank Dr Mike Tomson (APD) and Steve Duffy (Lay Assessor) for their critique of this document.
So where to start? Let’s look first at ARCP Panels, Educational Supervision and Work Place Based Assessment.
The Deanery website has useful information on ARCP Panels at
and Educational Supervision at
It states “The ARCP is the formal process by which a GP StR’s progression through training is reviewed and assessed. The ARCP panel of assessors confirms progress in each of the specialty training years and approves progress from one year to another. The final ARCP panel at the end of training will make a recommendation for the award of a Certificate of Completion of Training (CCT). This recommendation will be based upon evidence that the GP StR has covered the GP curriculum, including satisfactory workplace based assessments,the completion of all components of the new MRCGP exam and the reports of both the Clinical and Educational Supervisors. The Panel Chair will sign off the ARCP form for approval by the RCGP.“
Regarding Educational Supervision it summarises “the meetings are there to help the learner by giving them feedback, keeping them on track for their training and helping them to develop in those areas where there is an identified need.”
Work Place Based Assessment (WPBA) forms the third part of the MRCGP examination; the other two elements being the Applied Knowledge Test (AKT) and the Clinical Skills Assessment (CSA). There is real confidence in these two external exams. The way in which ARCP Panels look at WPBA has evolved over time and increased rigour and consistency is emerging. .
There are national requirements for WPBA for all specialties. These are defined by the Gold Guide 2010
Over time requirements for Y&HD Naturally Occurring Evidence (NOE)
including statements of Complaints, Half Day Release Attendance and Leave, plus Case Based Discussion (CbD) and Consultation Observation Tools (COTs) “grids” have been added. These elements have been included to help a GPStR demonstrate competence.
The Role of the Lay Assessor seems to have been less well defined. As I understand it they provide external scrutiny of the ARCP panel process. They need to clarify any conflicts of interest between the Educators and the GPStRs and provide their view on the information provided from the perspective of the lay person. To do this they need a detailed understanding of the e-Portfolio and what ARCP panels are looking for and this is the aim of the training session itself. They provide an invaluable contribution to the panel process.
Now have a look at the following links
and
We need to be able to understand the meaning behind the numbers and words recorded on the e-Portfolio.
Dr Adrian Dunbar has previously highlighted the danger of focusing on one or two smaller areas (the pixels) and thus losing sight of the bigger picture (the full screen). There are multiple sources of evidence on the e-Portfolio, more than ever used to exist in the past, so let us have a look at some of them in turn.
Posts
You need to know the whereabouts of the GPStR in their training. Are they coming to the end of their final year (ST3) and heading for their Certificate of Completion of Training (CCT) or are they progressing from their first year (ST1) to their second year (ST2)?
It is important as when considering recommendations for CCT you need to consider whether any concerns are so serious as to need say 6 months extra in training at a minimum cost of £40,000.
At earlier stages there can be no debate about Y&HD NOE including statements of HDR attendance, leave, complaints etc. Remember these elements enable a GPStR to demonstrate a range of competences. This is therefore to be considered in terms of “Development of Specific Competences” rather than “Missing Evidence” until completed at ST1 or ST2 progressions.
CCT however is assessed against National Requirements so whereas the Y&HD is able to strongly encourage people to present the Y&H Naturally Occurring Evidence they are unable to require their presence per se; but the panel is able to insist that there is sufficient demonstration of competence.
Scheme requirements for WPBA “Grids” for COTs and CbDs have recently been introduced. They should be being used formatively in the process of Educational Supervision to enable the GPStR to address areas in which competence has yet to be demonstrated.
A similar argument may apply where there is a degree of concern over the quality of Log Entries, PDPs, reflections on post, or “engagement with the e-portfolio” generally. You are allowed to be sad and disappointed, perhaps even irritated or angry, but is an extension to training warranted because of concerns in these areas alone?The assessor’ role is to establish the facts, chose representative examples and present this to the ARCP panel for a decision.
:LTFTT ^ Out of Sync
Then there are Less than Full Time Trainees (LTFTT). They are commonly referred up to central panel rather than be managed locally. This is because they are so complicated. The RCGP website details their requirements for evidence through the more information ...link at
Another grouping are the “Out of Sync” GPStRs. They may have been on Maternity or Sick Leave so their ST year end differs to their peers. Separate ARCP panels are organised often centrally.
Combine Out of Sync GPStRs going to LTFTT and you may struggle to understand what they need in place and by when. The best way to get your head around it is to look at where they should expect to be by that point from the start (or working backwards, before the end) of training.
You will need to become familiar with the Minimum evidence requirements for Full Timers and the Minimum evidence schedule for LTFTTs training at 50% of full-time. Then you can start to make your assessment.
Evidence
When ARCP panels first started up the focus was on the GPStR having the minimum evidence set in place. Though the way e-portfolios are reviewed has evolved a GPStR will always be found Unsatisfactory if the evidence is incomplete.
If one or two assessments or a piece of NOE are “missing” from the portfolio then the local or central panel may defer submitting a recommendation. They will agree a time to review the e-Portfolio again allowing a couple of weeks or so for the GPStR to get everything in place and then have a look at the portfolio again.
In practice this falls to the Panel Chair who signs them off as Satisfactory if they have everything sorted out properly. If not then they will be rated Unsatisfactory- Missing Evidence. This usually speaks of poor organisation on the part of the GPStR, but occasionally may represent a professionalism issue.
COTs or Mini-CEX
The Consultation observation tool (COT) is used to look at recorded or directly observed consultations and to make a comparison against the criteria described in the COT: Detailed Guide to the Performance Criteria.
Ratings are made against individual criteria and overall. There are grids available from the website which may be used by the ES to direct the GPStR to particular criteria Needing Further Development(NFD) or addressing areas where rated Insufficient Evidence (IE).The range of assessment outcomes are likely to be extended from August 2010.
Please remember a GPStR is of concern if there is a repeated pattern of NFD on an ongoing basis despite developmental feedback. If they have failed the CSA then this provides useful contextual information though we are encouraged to consider this separately from WPBA.
Indeed the odd NFD may be present in ST3 in a high flying GPStR who may have already passed the CSA and is being encouraged to bring along more challenging cases where there may be a chance to stretch them. Hopefully the trainer will have highlighted this in their write ups. The odd NFD therefore should not be a significant concern for CCT.
If a GPStR is allowed to regularly bring along consultations with a low degree of challenge such as a simple pill check or a mild viral illness then they may not be able to provide enough evidence of competence in the COT criteria.
The Clinical Evaluation Exercise(Mini CEX) is undertaken in hospital posts in place of a COT. They should be supervised and recorded by experienced doctors familiar with the procedure across a range of clinical problems. They focus more on Clinical Skills than Consultation Skills and the write ups often provide limited detail and so they are often difficult to assess in terms of what they mean.
Please remember when looking at Mini-CEX that many hospital consultants give ratings based on their assessment of a GPStR at that stage of training rather than judging them against a GP fit for independent practice. Many are therefore given inappropriately high ratings.
In Innovative Posts where the GPStR is split across GP and hospital posts the evidence set will include a mix of COT and Mini-CEX to the total stipulated for the post on the e-portfolio.
CbD
The Case-based discussion (CbD) is used to look at cases chosen by the GPStR recording the evidence provided against the appropriate competence area. Again grids are available from the website.
I would draw your attention to the following from the RCGP website information on CbD:
“It is unreasonable to expect that all the competences will be covered in a single CbD but if too few are considered useful evidence will be overlooked and there would be inadequate sampling of all the competences. It is helpful to tell the GPStR at the beginning of the discussion which competence areas you expect to look at.”
It may be useful to see the grids as helpful in a formative developmental sense; directing the GPStR on those areas rated NFD or IE as they progress. However they should not be used as a reason to give an Unsatisfactory Recommendation in assessing progress to CCT provided there is definite evidence of competences the grids suggest IE or NFD elsewhere on the e-Portfolio.
PSQ
The Patient Satisfaction Questionnaire will be used only once if the GPStR is in general practice for 12 months (in ST3) but twice if they have more than 12 months in general practice.Most GPStRs now have more than 12 months in GP placements. It used to be one of the commonest bits of Missing Evidence until this requirement was fully understood.
Mean, median and range scores are provided and the GPStR may be compared with their peers. Only numbers are provided; there are no written comments and numbers do not tell you THE STORY. s. Remember a doctor who gives a patient what they want may be rated more highly than a doctor who challenges a patient as to what is really needed but scores significantly below those of their peers should be considered alongside the other feedback provided from staff, doctors and supervisor/trainer..
MSF
Multi-Source Feedbackis undertaken with Two cycles to be completed in ST1 (5 clinicians only) and two cycles in ST3 (5 clinicians and 5 non-clinicians).
Beware the GPStR who upsets that barometer of opinion, the practice receptionist!
Word pictures are created and it is always worth reading in detail for concerns regarding performance, attitude, behaviour, teamwork, probity, engagement, reflection and learning etc.
CSR
The Clinical Supervisors Report was originally intended for the clinical supervisor to write a short structured report on the GPStR at the end of each hospital post. In the Y&HD it has also become the place where each GP Trainer is able to describe their GPStR’s progress.
The difference from the ES review, where they are being assessed in terms of fitness to practice independently at the end of training is that they are being rated in comparison to their peers i.e. how they are doing for their current stage of training.
BAgain, because specific feedback relating to the main headings of Relationships, Diagnostics, Management and Professionalism is provided from the Clinical Supervisor or Trainer it is o ne of THE most useful places to find useful information on the GPStR especially if comments are provided in addition to ratings.ne of THE most useful places to find useful information on the GPStR especially if comments are provided in addition to ratings.
Progress to Certification
Having reviewed the different sources of information on the e-portfolio for yourself it is now worth considering how the Educational Supervisor has assessed the GPStR’s portfolio.
If there is no up to date ES review then you may be unable to look at Competence Ratings and Curriculum Coverage etc and will need to ask your local Y&HD administrator, responsible for organising the ARCP panel, to prompt the ES to complete and submit the review (they may have saved it instead) or to undertake a new one if needed. An up to date ESR should be available to the panel and this should be completed no more than 6 to 8 weeks ahead of ARCP panel.
For West Yorkshire, N&E Yorkshire & N Lincs: email Lyndsey Pearson at and for South Yorkshire: email Janet Bell on
Sometimes GPStRs fail to contact their ES and arrange a meeting in time. It is the GPStR’s responsibility to do this; is their e-portfolio, their evidence etc and reflects on them.
There is a Quality Assurance programme which provides feedback to the Educational Supervisor. The form is downloadable on the ARCP panels page of the Y&HD website at
If you are asked to complete an ES feedback form you are being asked to assess the following:
Judgements are generally referenced to the available evidence
Judgements appear to be justifiable
The current state and the progression of competence are made clear
Suggestions for trainee development are routinely made and appear to be appropriate
You will also be asked to describe any Highlights or Suggestions for Improvement.
These criteria have been developed by the RCGP and Deaneries and Schemes are being judged against them.
Remember as always with feedback that you need to remain descriptive about specifics, sensitive to the person and their agenda, balanced in terms of positives and negatives and the degree of challenge needed to influence change.
Mandatory Elements
This should be easy. It is the word mandatory that gives you the clue…
It encompasses the external exams which must be passed plus accreditation in resuscitation and a sign off for OOH. There should be no real problem for the organised GPStR in getting the latter two signed off but the external exams require real application to ensure a good pass mark.
AKT
The RCGP state that the Applied Knowledge Test (AKT) enables candidates who have passed this assessment to have demonstrated their competence in applying knowledge at a level which is sufficiently high for independent practice.
Like the CSA, failure to pass the exam by the end of ST3 means that an extension to training is invariably granted, though the period of this extension may only allow for one further sitting.
The RCGP have recently advised that any GPStR starting their training after August 2010 is only to be allowed a maximum of 4 attempts. RCGP statistics indicate that anyone failing to pass after four sittings is unlikely to do so regardless of the number of further attempts.
You may wish to look at their mark and compare this with the pass mark and the mean mark.
CSA
The RCGP state that the Clinical Skills Assessment (CSA) is ‘an assessment of a doctor’s ability to integrate and apply clinical, professional, communication and practical skills appropriate for general practice’.
Like the AKT, failure to pass the exam by the end of ST3 means that an extension to training is normally granted, though the period of this extension may only allow for one further sitting.
The RCGP have recently advised that any GPStR starting their training after August 2010 is only to be allowed a maximum of 4 attempts for the same reasons given for the AKT.