Skills for Health Pre-Employment Programme Project
Evaluation Report, produced by Skills for Work
June 2010 – March 2011
Comments from programme participants and partners
‘Before I started this programme I was doing nothing and
I was a very shy person, but I have gone through this course,
I have built up a lot more confidence and
I am now more enthusiastic and motivated to work than ever before.’
‘For me it couldn't be improved as it taught me a lot of things,
motivated me, inspired me, introduced me to a lot of good people
who helped me try to do something with my life, which I'm forever debted for.’
‘It was useful to get in a good routine - getting out of bed early and
needing to be somewhere at a certain time again.’
‘I wasn’t just reading and learning about it I was doing it and
learning actually in the placement and I prefer it that way.’
‘I have now got an understanding of the role of a cleaner and
how important they are to the running of a hospital.’
Participants
‘We feel as a Trust it has been extremely worthwhile,
their enthusiasm to work was very evident…,
I'm sold on the programme!’
‘This programme puts them in front of the ‘competition’.’
When can we have them? Are they ‘ours’ now?
Trust managers
‘An excellent toolkit for the pre employment training programme.’
‘The fact that there was a potential job at the end
was a key factor in the success of the course
as it gave the participants a real reason to be committed.’
‘Now they’ve got positive references.’
Providers
Contents
Page
Evaluation Executive Summary 4
Section 1 Background 6
Section 2 Evaluation methodology 8
Section 3 Key findings 10
3.1 Feedback on Units developed
3.2 Learning outcomes
3.3 Learner reaction
3.4 Potential impact on job behaviours
and competences
3.5 Impact on organisational performance
3 .5 Stakeholder feedback
Section 4 Conclusions 25
Section 5 Recommendations 29
Section 6 Acknowledgements 31
Appendix 1 Evaluation Framework
Appendix 2 Learner Reaction, Furness College
Appendix 3 Learner Reaction, Lancaster & Morecambe
College
Appendix 4 Unit feedback, summary
Appendix 5 Learning Outcomes, Furness College
Appendix 6 Learning Outcomes, Lancaster & Morecambe College
Appendix 7 NHS Managers’ Feedback, Furness General
Appendix 8 NHS Managers’ Feedback, Lancaster Royal
Infirmary
Appendix 9 Stakeholder Survey
Evaluation Executive Summary
The Skills for Health Pre-employment Programme Pilot has been an outstanding project. The pilot has been characterised by exceptional commitment from all concerned. Partners included Skills for Health, Jobcentre Plus, STEPS, Skills for Health Academy, North West, University Hospitals of Morecambe Bay NHS Foundation Trust, Lancaster and Morecambe College and Furness College.
Training materials developed have been aligned to key health sector employability and career frameworks (NHS Knowledge and Skills Framework and Sector Qualification Strategy and Employability Skills Matrix, which are mapped to the Career Framework for Health) providing value to participants, providers, and employers.
The programme materials and guidance were regarded to be of high quality, interactive, and fit for purpose.
Generally programme participants found the classroom teaching interesting and enjoyable; classroom teaching is not usually popular for this client group. The most significant outcomes of the project for the 17 long-term unemployed young people were the development of employability skills, self esteem and positive future work prospects in the health sector. 16 participants were recruited to the NHS staffing Bank, and 5 participants were recruited to actual jobs within the Trusts within approximately one month from the end of the programme. Most participants were keen to consider further training or higher education and careers within the health sector at the end of the programme.
The prospect of an interview with a real job opportunity at the end was crucial to participants.
Improving employability and providing realistic job prospects for long term unemployed individuals is complex work requiring effective collaboration amongst a number of agencies. Strong partnership working, an understanding of the client group, early involvement of JCP Advisers, excellent materials, experienced providers, ‘hands on’ coordination, intensive mentoring and coordination and a supportive employer were key components of the success of this programme.
Pilots were run at 2 sites operating differently, providing useful, broader evaluation and indicating two work placement models.
Comments from the participants, managers and providers were positive in all important respects.
Some useful feedback was provided on minor details of the materials that need to be adjusted. Some adjustments to the guidance materials are also need to highlight features of the pilot that were key to success.
The analysis of expected return on training investment for this programme, based on 16 new recruits to the NHS Bank, actual costs and conservative estimates of financial benefits, indicated a good return on investment (122.%).
Where several cohorts were to be recruited over a year period through pre-employment programmes, this benefit would be multiplied and could add up to a significant annual saving.
Publicity materials need to highlight the value and benefits of the programme as noted in the conclusions and recommendations of this report.
Evaluative feedback from this pilot strongly indicates that it would be worthwhile to undertake additional development to create a health sector pre-employment qualification.
Section 1 - Background
Skills for Health (SfH) and Jobcentre Plus (JCP) have developed and piloted a pre-employment training programme, which builds on the existing Skills for Health Employability Toolkit (SET). This programme was commissioned by the Department for Work and Pensions (DWP) developed in partnership with the JCP National Skills Partnerships team. It was designed for entry level Support Worker roles, to support the following groups into employment and to progress within the health sector:
§ primary audience, young people (18-24 years) as part of Routes into Work contracts
§ long term unemployed
§ a wider JCP target group.
The programme was designed to contribute to recruiting and growing the workforce for the future by providing a progression pathway which can lead to Level 2 qualifications and Apprenticeship Frameworks appropriate to the sector and is mapped to the NHS Knowledge and Skills Framework, Core Dimensions at Level 1.
In order to have the maximum value to participants and health sector employers the programme aimed to align to Skills for Health Qualification Strategy and Employability Skills Matrix, which are mapped to the Career Framework for Health. The programme also aimed to map to other relevant qualification frameworks at Entry 3/Level 1, as well as signposting to additional support for Functional Skills - English, Mathematics and ICT.
Skills for Health developed the programme in partnership with Jobcentre Plus (JCP), STEPS, Skills for Health Academy, North West (SfHA-NW), University Hospitals of Morecambe Bay NHS Foundation Trust (MBHT), (2 sites: Lancaster Royal Infirmary and Furness General Hospital). Lancaster and Morecambe College and Furness College (both Routes into Work contractors) delivered the programme at the respective sites. Local JCP Advisers supported by the training providers assisted the pilot site employers to recruit to hard to fill, skills shortage entry level jobs. SfHA-NW Pre-Employment Leads acted as brokers and intermediaries with all partners at local level. Service Level Agreements were put in place between the SfHA-NW, the employer and training providers.
Overview of the programme
The Pre-employment Training Programme was designed to provide a consistent approach towards helping unemployed people into sustainable employment in the health sector. The programme provided:
§ employability skills
§ introductory sector specific training
§ work placement.
The programme developed was modular allowing for delivery over 5-8 weeks, depending on the needs of the employer and of the programme participants. The pilot ran for five weeks with work placement built in. Participants’ individual needs were assessed at selection/recruitment and they were signposted to additional support from providers so that individual needs could be catered for.
The Pre-employment programme was designed for delivery to participants who had no or few qualifications, and who were assessed as having literacy skills at Entry Level 3 or Level 1.
There were 36 Units developed of which 24 were designate as Core Units and a further 12 as Optional Units. However Units were designed to be selected flexibly to meet the needs of the particular participants and work placements.
Section 2 - Evaluation Methodology
Skills for Health (SfH) selected Skills for Work to undertake the external evaluation of the programme pilots. Skills for Work are specialists in training evaluation for entry level jobs and also recently developed (with the support of EACEA, Leonardo da Vinci funding) an online toolkit (TrainingEvaluationPro) to evaluate the impact of all types of training.
SfH was keen to trial the use of TrainingEvaluationPro including to measure the impact of the programme in relation to key Hospital Trust performance indicators (KPIs) such as employee turnover, sickness/absenteeism, and reductions in induction costs. The project started some months later than originally expected so it was not possible to conduct a return on training investment (ROI) evaluation. However MBHT agreed to undertake an evaluation of expected return on training investment which could be followed up between 3 to 12 months later by an actual ROI analysis.
An Evaluation Framework, setting out the evaluation measures and the process for collecting and presenting data, was prepared, see Appendix 1 and circulated for comment amongst members of the project Virtual Reference Group.
The evaluation focused on the design, delivery and impact of the programme on both individuals and the employer organisation. Evaluation of the programme, supporting materials and work placements followed the four level Kirkpatrick evaluation methodology including:
§ learning outcomes
§ learner reaction
§ potential impact on employability skills
§ potential impact on organisational performance and expected return on training investment.
Evidence of potential impact was based on the judgement of stakeholders.
Project level evaluation was undertaken although this has only been reported on where feedback shed light on the conditions that were necessary to run the programme successfully. All project partners completed surveys on the following aspects of the programme:
§ partnership working
§ fitness of the programme as route way into employment
§ sustainability of the programme.
Evaluation criteria and survey questions were drafted and circulated for comment by SfH, SfHA-NW and the both providers. The Evaluator met with providers and the Hospital Trust to finalise the evaluation surveys. Expected ROI criteria were agreed with SfH, SfHA-NW and the MBHT.
The evaluation of the project and the programme was conducted during and after programme delivery.
Qualitative and quantitative evidence indicating programme value and potential impact was gathered from project stakeholders, including reference group members, participants, programme developers, the Hospital Trust, training providers and Job Centre Plus.
Evaluative data was also gathered from:
§ minutes of Virtual Reference Group meetings
§ online surveys
§ 1:1 interviews with participants conducted by SfHA-NW coordinators
§ evaluation focus group at the end of the project
§ end of programme certificate presentation and celebration events at both Furness and Lancaster.
Day by day feedback provided on Units delivered enabled project managers and materials developers to support delivery teams where there were a small number of queries on delivery.
Statistical significance
There were 2 pilots over 2 different sites which enabled comparisons between training delivery and work placement experiences.
The small size of the groups of participants (8 at Lancaster and 9 at Furness) has been taken into account when analysing significance.
Section 3 - Key findings
Detailed evaluation survey results are available in the following reports:
§ Learner Reaction, Furness College (Appendix 2)
§ Learner Reaction, Lancaster & Morecambe College (Appendix 3)
§ Learning Outcomes, Furness College (Appendix 5)
§ Learning Outcomes, Lancaster & Morecambe College (Appendix 6)
§ NHS Managers’ Feedback, Furness General (Appendix 7)
§ NHS Managers’ Feedback, Lancaster Royal Infirmary (Appendix 8)
§ Stakeholder Survey (Appendix 9)
Data on Programme participants
Participants / Combined / Furness / Lancaster & MorecambeNumber of participants / 9 / 8
% of participants completed / 100% / 100%
Completed evaluation survey / 8 / 8
Age 18 - 24 / 100% / 9 / 8
Male / 77% / 8 / 5
Female / 23% / 1 / 3
First job / 56% / 50% 4/8 / 62% 5/8
Level of participants in relation to the course materials
Pilot participants all undertook Initial Assessments in Literacy and Numeracy as part of the Selection and Recruitment process, and the small number of participants who needed to develop those skills were signposted to additional support.
These two cohorts of participants had relatively high literacy levels in relation to the target group for the programme. It is therefore important to interpret the feedback provided below accordingly.
3.1 Feedback on Units developed
Unit Feedback has been summarised, see Appendix 4.
3.2 Learning Outcomes
The following learning outcomes were achieved:
Cohort / Furness / Lancaster & MorecambeCertificates achieved / § First Aid
§ Manual Handling
§ Food Hygiene
§ Health and Safety / § First Aid,
§ Manual Handling,
§ Royal Society for Public Health level 2 Award in Food Safety in Catering
Other skills gained / § Communication
§ Employability
§ Equality & Diversity
§ Confidence
§ Making a presentation
§ Working with different types of people
§ Improved self esteem
§ Renewed enthusiasm for work and life in general
§ Giving up smoking
§ Made friends for life
§ Worked as part of a team
3.3 Learner Reaction
Responses from each cohort of participants were reported separately see Annex 2 and Annex 3. Their comments were broadly similar and have been summarised together below.
Reasons for taking part in the training programme
Course participants indicated that they were well motivated to attend the course from the start:
§ Being out of work was demoralising and I was beginning to lose hope to get back in.
§ It was a good opportunity to get back into work life and hopefully gain employment.