Portsmouth City Teaching PCT APPRAISAL SERVICE

NESC

Portsmouth City teaching PCT APPRAISAL SERVICE

END OF YEAR REPORT

Summary

• 2008/9 was the first year that NESC was contracted to provide an appraisal service for Portsmouth City tPCT.
• In a shortened appraisal year, NESC successfully appraised 143 GPs in Portsmouth City tPCT. This represents 100% of all eligible GPs. 11 Portsmouth appraisers were contracted to enable this, and 12 external appraisers from within NESC supplied the additional capacity that was needed.
• NESC has provided evaluation, support, and training of the appraisers, and is well prepared for revalidation.
• Learning needs have been collated for GP tutors and primary care educators
• The appraisal process has been comprehensively evaluated for quality assurance and participating satisfaction. The results of the evaluations show a much higher degree of engagement and appreciation of appraisals than has been reported nationally.
• A local appraisal lead for Portsmouth (Dr Sue Bowen), an extremely experienced appraiser, has been successfully appointed to take appraisal in Portsmouth forwards in 2009/10
• An active recruitment and training programme has ensured that for 2009/10 there will be sufficient capacity among Portsmouth appraisers for all Portsmouth appraisees.

Background

In April 2008 the NHS Education South Central (NESC) was awarded the contract for delivery of the appraisal service by Portsmouth City Training PCT. The first meeting with potential appraisers took place in June 2008. A total of 10 appraisers accepted our terms and conditions and were offered honorary contracts. A further Portsmouth appraiser was trained and appointed during the year. We were then able to commence the process of allocation. As the annual appraisal year end is 31st March, this allowed 9 months to arrange and deliver appraisals to the 172 GPs who appeared on the PCT’s performers’ list.

Attempts to engage a local appraisal lead GP on two occasions were unsuccessful as none of the existing Portsmouth appraisers felt they had the capacity to take on such a commitment. As a result, as an interim solution, Dr Susi Caesar, the overall NESC Appraisal Lead, took on the responsibility. In addition, Dr Eileen Gorrod, an appraisal lead within Hampshire PCT undertook some appraiser support group work with the existing appraisers, and Dr Betty Charlton, a Portsmouth appraiser, took on the analysis of the Form 4s and PDPs arising from the appraisals. An administrator was engaged to manage the process, under the guidance of Mrs Beryl Hodgson, the NESC Primary Care Taskforce Programme Manager.

Performer's List management and appraiser matching

The first task was to allocate GPs from the performers list to appraisers. It became clear very quickly that the performers list held by the PCT and SHA was inaccurate. It included doctors who had moved away or abroad, had retired or did not require an appraisal for other reasons. This necessitated some collaborative investigations in the early stages of the process, correcting the list, tracking those who needed appraisals and allocating them to appropriate appraisers. NESC kept in close communication with the PCT updating the performers’ list with changes to it. This appears to be one of the unintended benefits of contracting for appraisal services with NESC, as other PCTs have had similar issues with maintaining the accuracy of their performers’ lists. It is hoped that with ongoing collaboration between NESC and the tPCT, we now have a more accurate list for the next appraisal year.

There is also a subsection of GPs who are correctly on the performers list but who are excluded from the contract requiring NESC to provide an annual appraisal. These included GPs employed by other agencies i.e. MOD, those on long term sabbatical, sickness or maternity leave, prison doctors, and GP registrars just out of training whose appraisal had been completed by their GP Trainer. All these doctors, though not needing appraisals, still required NESC administrative support and monitoring.

The final total of GPs that required appraisal was143.These were randomly allocated to an appraiser who was not a GP Partner or otherwise known to be linked to the appraisee, using information about who had previously appraised them where available. The appraisers were given the opportunity to request a reallocation where appropriate. Each GP was informed in writing about who their allocated appraiser was and they also had the opportunity to request a change of appraiser. All those requesting it were reallocated.

Although this was a new system for Portsmouth appraisers and appraisees alike, it avoids any possible appearance of collusion and is considered more robust than an appraisee choice system. There were some initial concerns expressed as the previous system in Portsmouth had been well respected, but overall the change was well accepted.

Appraisal Delivery

A total of 142appraisals were delivered by 31st March 2009 by 23 appraisers. (10 original Portsmouth appraisers, 1 new Portsmouth appraiser trained in year and 12 supplied by NESC). One doctor who had only appeared on the performer’s list in February 2009, was granted leave by the PCT to have an extension (in order to have the appropriate notice period and time to prepare) and was appraised on 17.04.09, but is included in 2008/9 figures, bringing the total appraisals delivered to 143. (100% of eligible appraisees).

This is a tremendous achievement and far exceeds that of many other PCTs. The PCT has been informed of those that did not require appraisal, with explanations. Communications with the PCT were most fruitful in organising the last few appraisals.

After each appraisal the appraiser sent a copy of the appraisal summary (Form 4) and the appraisee’s PDP to NESC. The appraisee was emailed and asked to complete an evaluation of the appraisal and return this to NESC. These documents are stored securely by NESC, filed by appraiser and accessible only to the local appraisal lead, (or in 2008/9, Dr Betty Charlton, in lieu of a local lead) the NESC Appraisal and Revalidation Lead (Wessex) and the appraisal administrator.

The security and confidentiality of the documents held is of paramount importance to NESC and all systems are in line with national guidance.

Appraiser Support

GP appraisals are peer led, formative, developmental and used to encourage and challenge better practice. As such they are very different from many annual performance evaluations that take place in other organisations. It is important that they must be done well and the skills needed are complex and multi faceted. Previously appraisers have worked in isolation from each other with little or no feedback about their appraisal work.

Since the contract was awarded the appraisers have been given formal and informal support throughout by the NESC team approach. All Portsmouth appraisers were offered the opportunity to take part in the NESC wide existing appraiser skills assessment programme and other educational events including a whole day refresher event, held 4 times throughout the NESC area, including in Portsmouth. A local support meeting was held by Dr Eileen Gorrod for all the appraisers to offer peer support and exchange ideas and views. At the early meetings appraisers were keen to understand the NESC processes and to know how they were doing and wanted to have feedback on their appraisal skills.

NESC also has a requirement to ensure that the appraisal process is being undertaken to the highest professional standards.

There are some national guidelines as to what appraisals should contain. We decided to extend an in depth analysis of appraisal paperworkwhich had been piloted in Dorset PCT the previous year to both assure quality and for training purposes. This was well received by the Portsmouth appraisers.

Post Appraisal Quality Assurance

The quality of the appraisal process and the skills of the appraisers have been analysed in three separate ways. This is to triangulate the evidence and hopefully gain better information about the quality and outcome of appraisals and also to inform NESC about the individual learning needs of the appraisers.

The three approaches to measure quality of appraisals were:

1Appraisee Evaluation: each appraisee is asked to complete an evaluation of their appraisal and return it to NESC. This asks them to score and comment on four areas of the process:

• The Appraisal Organisation
• The Appraisal meeting
• The Appraiser
• An Overview of Appraisal

Of the 143 appraisals carried out 83evaluations (58%) were received. The summated comments are attached as a spread sheet.

2Quality Assure the Form 4:each appraiser’s paperwork i.e. the Appraisal Summaries and PDPs were analysed and marked against a scoring grid looking for elements considered important. This grid (LEARN IT) is attached. It was developed by NESC in Dorset with reference to other publishedwork.

At the same time obvious learning needs identified and recorded in the appraisal were noted.

3Appraiser Self Rating: Each appraiser was asked to complete a self rating scale on their own perception of the areas of weakness and strength

Appraiser END OF YEar Feedback

Each appraiser is being contacted by the newly appointed local appraisal lead to review their personal results. They will have individual written information about the summated evaluations from their appraisees, the quality assurance rating of their paperwork and their own self rating form. All appraisers will be given the opportunity to discuss the relevance of these results with their peers at evening meetings. There has been genuine interest from the appraisers in receiving feedback and defining their own ways of improving in the appraisal part of their overall personal development plan.

Results

Overall the results show a very high standard of appraisee satisfaction with the process. The quality of appraisers is also generally very high and we did not discover any significant difficulties with performance amongst the appraisers. However, the evaluations did highlight some areas for improvement and also a few individual needs that will be addressed with the respective appraisers.

Some interesting points from the evaluation responses we received:

80% of appraisees felt that the appraisal was useful to their professional development.

91% felt the appraiser was prepared and listened to them.

62% did the appraisal in time that was not protected.
76% felt challenged by their appraiser.

91% felt supported by their appraiser.

The statistics are well in excess of the levels reported elsewhere nationally, and in line with those reported in Dorset in the first year of working with NESC. They reflect a commitment to high quality formative appraisal from the PCT both now and in the past, as well as good appraiser training. However, 58%, though well above some nationally reported return levels, is lower than we would anticipate at NESC and is an area we will be addressing with appraisees in 2009/10.

Ongoing Training and Skills evaluation

NESC ran a series of educational events,open to all NESC appraisers, throughout the year looking at different aspects of appraisal, including new appraiser training, existing appraiser skills assessment, where mentoring meets appraisal, writing better Form 4s and the annual refresher day. While there was an intention to hold 4 local support group events per year, this was neither appropriate in the shortened time frame, nor achievable in the absence of a local lead, and only one meeting of this type was held in 2008/9. NESC recommend that each appraiser must attend 6 hours of continuing professional development as an appraiser per year, and must also have a full day competency based skills assessment every 3 years. Of the existing Portsmouth appraisers, all achieved their CPD targets.In addition,5took part in competency based assessment. This figure would have been higher if it not been for the cancellation of a competency based assessment day due to snow in February, and the 3 Portsmouth appraisers who were scheduled to attend that event, were eventually assessed on 21.04.09, bringing the total who have voluntarily undergone assessment in the first year of the three yearly cycle to 7 of the 10 existing appraisers (70%).

COMPLAINTS

There were no complaints about the Portsmouth appraisal service received in the year 2008/9.

Conclusions

After a late start for both the tPCT and NESC,fully 100% of eligible GPs have been successfully appraised in 2008/9.

A careful and detailed evaluation of the quality of appraisals delivered has been undertaken, both internally and externally. We are proud to highlight the findings of the external (AQMAR) review team:

• "the initial and ongoing training and support of appraisers within NESC are exemplary"
• "NESC is innovative in introducing appraiser performance processes"

The results showed a higher level of satisfaction amongst GPs with their appraisals than reported nationally. We believe this reflects not only the quality and skills of local appraisers but also the approach and support of the tPCT.

The audit reports are being used in individual feedback to appraisers and anonymously for general training purposes.

Sufficient new appraisers have been trained for Portsmouth to have local appraisers for all who need them.

A local appraisal lead has been appointed for 2009/10

Feedback from the results of the external quality assurance will continue to drive improvement in appraisal in 2009/10

Dr. Susi Caesar

Appraisal &Revalidation Lead (Wessex)

Attachments:

1ACollated Appraisee responses

Total Number of Appraisees:83

Cumulative Preparation Time:837.5hoursAverage:10.5hours

Cumulative Appraisal Time:195.25hoursAverage:2.35hours

Appraisal Organisation

Yes / No
Were you given adequate notice to allow you to prepare for the appraisal meeting? / 75 / 1
Did the information from the Deanery tell you all that you needed to know about your appraisal? / 63 / 12
Did you receive adequate support to allow you to prepare for the appraisal meeting? / 59 / 9
Was your preparation time protected? / 26 / 52
Did you know where to find the appraisal documents and forms? / 70 / 7
Did you use the NHS Toolkit in preparing for your appraisal? / 47 / 30
Was the venue for the appraisal meeting appropriate? / 76 / 1
Was the agreed date and time of the meeting convenient for you? / 75 / 0

The Appraisal Meeting

Yes / No
Were the goals of the appraisal meeting clear? / 77 / 0
Did you have enough time to discuss all the issues that were important to you? / 79 / 1
Did your appraiser reflect on your previous year's appraisal outcome? / 77 / 2
Did you feel that the time spent on the appraisal discussion was worthwhile? / 77 / 2
Did you feel at ease during the meeting? / 79 / 1
Was the appraisal discussion constructive? / 76 / 2
Was agreeing goals and PDP activities straightforward? / 79 / 0
Do you feel the goals agreed in the appraisal summary are achievable? / 80 / 0
Does your development plan reflect your main priorities for development? / 79 / 0
Would you have preferred any aspect of the meeting to have run differently? / 18 / 60
Would you have liked your appraisal meeting to have included any other areas not spoken about? / 14 / 66

My Appraiser

Yes / No
Did your appraiser discuss the content of the appraisal meeting with you beforehand? / 37 / 39
Did your appraiser appear to have prepared for the appraisal meeting? / 75 / 1
Did your appraiser listen to you? / 76 / 0
Was your appraiser challenging in his or her questioning? / 63 / 13
Was your appraiser supportive? / 76 / 0
Did your appraiser prompt you to think about new areas for development? / 69 / 7
In your view, was the appraiser skilled in conducting your appraisal? / 74 / 1
Would you be happy to have the same appraiser again? / 74 / 2

Overview of Appraisal

Yes / No
In your view is the appraisal process useful to your professional development as a doctor? / 66 / 8
Did the appraisal meeting cover all aspects of your role as a GP? / 73 / 3
In your view is time spent preparing for an appraisal, time usefully spent? / 63 / 10
In your view, will the agreed goals and PDP activities enhance your work as a GP? / 67 / 8

1bGeneral comments from Appraisees

Toolkit

“NHS Appraisal Toolkit very useful.”

“Unable to use NHS Toolkit - unable to access the correct year.”

“Difficult to use Toolkit first time.”

“Appraiser not trained to use NHS Toolkit so I had to print it out.”

“Although you recommend NHS appraisal Toolkit neither the appraiser or your admin staff trained to use it.”

“My appraiser was unable to access the Toolkit website for me. I needed to print off forms for her to see what I had put on the site.”

Things that work

“Very supportive in all respects, guides me for agreed goals for next PDP.”

“Also important to have "mature" appraisers when I am also ‘maturing’. “

“Helpful and objective. “

“Empathetic, easy to talk to, some useful ideas discussed.”

“Questionning [sic] appropriate.”

“Very supportive and helpful.”

“Pleasure to meet with Betty again.”

“Excellent, helpful and understanding.”

“Dr. Charlton is an excellent appraiser.”

“Most thorough appraisal yet - thus most useful.”

“Really helpful and constructive.”

“I think most GP's use the appraisal as being contributing to revalidation and therefore this adds to the supporting evidence submitted.”

“Appraiser was well prepared and supported me to think about my development needs.”

“Good to have an appraiser who is local and understands local problems.”

Possible Improvements

“Appraisal done in own time and if appraisal done during surgery hours we have to make up time.”

“Preparation mainly in own time or study leave, appears no other way of obtaining protected time, appraisal preparation time is constraining.”

“Too little space for additional comments on forms.”

“My appraisal is always carried out in August when two doctors on holiday making workload heavy for remaining partners.”