Guide to Evaluating Diversity Training
In the Fire and Rescue Service
By Dr Steven Wilkinson
For the FireWorks for Equality in the Fire Service Project
AngliaRuskinUniversity
A European Social Fund Project
Foreword
This Guide to Evaluating Diversity Training was developed as part of the Fireworks for Equality in the Fire Service Project, undertaken by AngliaRuskinUniversity (Anglia)
( Although this document has been designed with the Fire and Rescue Service (FRS) in mind, it also serves as framework for all practitioners and organisations involved in the delivery of Diversity training to reflect on their efforts and draw out valuable lessons.
We are committed to ensuring that everyone is aware of the benefits of a Diverse workforce, as well as ensuring that fair access to opportunities allows for all individuals to take their talents and interests wherever they might lead them. Evaluation of Diversity Training is one important piece of the overall strategy and is crucial if we are to learn how to embed change in our society effectively.
This Guide, recently created and piloted within the FRS, provides a key tool for Diversity Trainers, Educators and Practitioners.
Dr Jaki Lilly
Research Director
Fireworks Project
AngliaRuskinUniversity
East Road
Cambridge CB1 1PT
England 2005
1
© 2005, AngliaRuskinUniversity FireWorks Project
Contents
Foreword...... 2
Contents...... 3
Introduction...... 5
The aims of the Guide...... 5
Using the Guide...... 5
The development of the Guide.....5
Equal opportunities...... 5
Who is the Guide for?...... 5
What kinds of Diversity Training?...6
What is evaluation?...... 6
Why evaluate?...... 6
Should evaluation be internal or
external?...... 7
Pre-planning...... 7
Where to Start...... 8
The Model...... 9
How this guide is presented...... 10
Review...... 10
Training Needs Analysis – Review...10
Focus questions...... 10
Why Evaluate the Training Need?...10
Policies...... 10
Role Maps...... 12
National Standards...... 12
Workplace Training Needs Analysis..13
Impact Assessment...... 15
Outcome of Stage 1 - Training Needs
Analysis (Review)...... 19
Plan...... 20
Training Development – Plan...... 20
Focus questions...... 20
Why Evaluate the Training Plan?....20
Setting objectives or competencies..20
Objectives...... 20
Verb Test...... 20
Outcomes Test...... 20
Characteristic Test...... 21
Achievement Test...... 21
Objective or TeachingPointTest....21
Fit for Purpose...... 22
The Training Delivery Plan...... 25
Questionnaires...... 25
Constructing a Questionnaire...... 25
Post Training Survey...... 29
Outcome of Stage 2 – Training
Development Plan (Plan)...... 34
Deliver...... 35
Training Delivery – Deliver...... 35
Focus questions...... 35
Why Evaluate Training Delivery?....35
Evaluating the Effectiveness of
Delivery...... 35
Learning Styles...... 35
Adult Learning Principles...... 35
Learning Events...... 36
Observation...... 36
Evaluating Learning Outcomes.....36
Self Evaluation...... 39
Method...... 39
Self Evaluation Questionnaire...... 39
Log...... 39
Combining Evidence...... 39
Evaluating Learning Outcomes.....42
Evaluating Training Materials...... 45
Outcome of Stage three - Training
Delivery – Deliver...... 47
Appendix 1...... 48
Assembling Judging Success and
Quality, interpret the evidence
and Writing the Report...... 48
When to assemble and interpret the
Evidence...... 48
What does the evidence tell us?....48
Judging success and quality...... 48
Possible measures of success...... 48
Interpreting Evidence...... 50
Using percentages...... 50
Interpreting qualitative evidence....50
Combining different kinds of evidence50
Meetings...... 50
Why have a review meeting?...... 50
Running a review meeting at the
end of an Evaluation Stage...... 50
Making changes...... 51
Meeting after the Evaluation Cycle
(all 3 stages)...... 51
Getting an overview...... 51
Learning from evaluation...... 51
How and when to reflect...... 51
Moving forward...... 51
Comparing courses...... 51
Using the evaluation to show
What happened...... 51
Telling people outside of training...51
Writing a project report...... 53
Common pitfalls of written reports..53
Suggested format for a written
evaluation report...... 53
Section headings...... 53
Context...... 53
Description of training...... 53
Aims, objectives and measures of
Success...... 54
What we found out...... 54
Overview of the project...... 54
Lessons learned from the evaluation.54
Appendices - Might include:...... 54
Report Framework...... 54
Appendix 2...... 55
Techniques for collecting evidence,..55
What kinds of evidence?...... 55
Choosing ways of collecting evidence55
Techniques for collecting evidence..56
Comments’ box...... 56
Display at end of training...... 56
Charts and diagrams...... 56
‘Graffiti’ walls...... 56
Individual interviews...... 57
Observation...... 57
Online websites, chat rooms
and email...... 57
Participatory techniques...... 57
Photography...... 58
Questionnaires...... 58
Process Questions...... 58
Analysis of the answers...... 58
Small group discussions...... 59
Tape recordings...... 59
Video...... 59
Written diary...... 60
Alternative sources of existing Data.60
Appendix 3...... 61
Evaluating E-Learning Programmes..61
Key People...... 61
Questionnaires...... 61
Constructing a Questionnaire...... 63
Appendix 4...... 66
Emergency Fire Services National Occupational Standards (NOS)
Knowledge and Understanding Requirements for Equal Opportunity & Diversity66
Acknowledgements...... 72
Tools
Policy Audit...... 11
Role Map and National Standards
Audit...... 12
Diversity and Equality Training Needs Analysis Survey14
Diversity and Equality Training Focus Group Meeting16
Objective Evaluation Form...... 23
Fit for Purpose Template...... 24
Training Feedback Questionnaire...27
Training Feedback Template...... 28
Post Training Survey Questionnaire..30
Post Training Survey Feedback
Template...... 32
Training Observation Schedule.....37
Trainer Self-Assessment...... 39
Log...... 41
Training Evaluation Test...... 44
Template for Evaluating Training Materials46
E-Learning Benchmarking Template.62
E-Learning Questionnaire...... 65
1
© 2005, AngliaRuskinUniversity FireWorks Project
Introduction
The aims of the Guide
This Guideaims to help everyone involved in Diversity Training to understand evaluation clearly and to evaluate effectively, according to their particular needs. In the long term, the aim of the Guide is to raise the standard of Diversity Training.
Diversity Training is difficult to evaluate and this Guide does not try to suggest that there is only one way of approaching evaluation. It aims to provide a flexible framework, which can be applied in many different situations and used to evaluate short or more extended training programmes. This Guideis for everyone; whether you are experienced in evaluation techniques or an absolute beginner, the Guide is designed to help you get the most out of the time you invest in evaluating training.
Evaluating the quality of training is open to argument and interpretation and the Guide suggests a range of measures of success for training, with the emphasis on flexibility and fitness for purpose.
This Guide may also be used to development a training program, starting from scratch.
Using the Guide
The Guide can be read in its entirety, or you can dip into it, to get advice on particular aspects of evaluation. The Guide is designed to be used in different ways. You do not need to evaluate every aspect of your training in-depth, but can use the Guide to help you focus on one or two areas, according to your needs. Many pages are photocopiable, and wherever possible key concepts, ideas and questions are highlighted so that they can be easily picked out.
The development of the Guide
The Guide was adapted from one developed by Felicity Woolf for the Arts Council England titled ‘Partnerships for Learning: A Guide to Evaluating Arts Education Projects’, Arts Council England, ISBN 0-7287-0791-8 © Arts Council England, first published 1999 and revised June 2004. The Diversity Training content was developed by Dr Steven Wilkinson after extensive consultation with Fire and Rescue Services, trainers, education consultants, academics and equality officers. It was piloted in draft form by 11 Fire and Rescue services throughout England and at dedicated development seminars and meetings. Many of the examples included in this Guide came from these pilots.
Equal opportunities
Evaluation should give everyone involved in Diversity Training the opportunity to contribute their views. No one should be excluded from this process because of their race, gender, sexual orientation, social background or disability.
Who is the Guide for?
The Guide will be useful to anyone who organises, funds, delivers or takes part in Diversity Training which aims to have learning outcomes.
What kinds of Diversity training?
The Guide is to help evaluate Diversity training in the context of lifelong learning, which aims to do one or more of the following:
- increase knowledge and understanding of Diversity and Equality in the workplace
- develop skills and performance to incorporate Diversity and Equality policy
- bring about personal and social change in line with the Diversity and Equality
agenda
In the Guide the term ‘Diversity and Equality Training’ includes:
- awareness and policy programmes for all members of the FRS
- specific training for Recruiters and/or Trainers
- diversity planning and policy development for Managers
- training and ongoing professional development for Equality Officers
- wider training programmes for which Diversity is one of the outcomes
What is evaluation?
Evaluation is a powerful tool for learning. It is a structured way of thinking about what happens during and as a consequence of training, and why. It can be simple or complex, depending on the resources you have available, and on what you want to find out. We suggest that evaluation is based on the following three key ideas.
- Evaluation involves making judgements, based on evidence, about the value
and quality of training.
- Evaluation is open and clear and involves a range of interested parties,
including the people taking part and people for whom the training is designed
to have impact upon.
- Evaluation helps with decision-making during training and for future training
events.
Why evaluate?
The purpose of evaluating training is to assess:
- how training is meeting the needs of the learners,
- the acquisition of new knowledge skills and attitudes,
- the impact of learning on the workplace, and
- the impact of learning on the organisation.
Improving is important so that:
- training content can evolve and change as necessary
- training can be better and standards raised
- trainers can change and develop their practice.
It is important to show:
- that training is effective in implementing the Diversity agenda
- the benefits of training at all levels
- that training is a cost effective exercise.
Should evaluation be internal or external?
The Guide focuses on helping organisations to evaluate their own training, but there may be circumstances when you would find it better to organise external evaluation. Even with an external evaluation, organisations should ensure that they are fully involved with the process.
External evaluation / Internal evaluationAdvantages
Time is allocated to the evaluation because it is paid for Could be seen as more objective by people outside, and so the conclusions may be more credible
May bring in new perspectives, widen discussion and increase learning
More likely to raise questions some will hesitate to bring up if evaluation is internal. /
Advantages
Probably cheaper in cash terms Creates ownership, which is more likely to lead to reflection and changes in practice
Stakeholders can both control the process and suit it to their needs
Stakeholders can avoid going too far with critical comments.
Disadvantages
Will add to training costs
Organisations have less control over the evaluation process and may have less
ownership of it
The process may be less integral to the project and ‘bolted on’
Exposing difficult issues and weaknesses may alienate some stakeholders. /
Disadvantages
Takes time and is likely to create additional tasks May lack credibility with outsiders
The evaluation may be narrower, without an external perspective
More likely to skirt round difficult issues or weaknesses.
Pre-planning
Good planning is vital for effective evaluation, whatever the scale of your training. You need to think about evaluation from the very beginning – before you commence any training, along with the other practical details. Evaluation needs to be programmed into your schedule, so that time is allowed for all the stages of the process. If you don’t think about evaluation until you commence training, you may miss important evidence or find it more difficult to collect.
You also need to include evaluation in the training budget, as it takes time and resources. For example, external trainers will require to be paid for attending evaluation meetings
At the planning stage, think about:
- Who will have overall responsibility for evaluation?
- Who else will have a role in collecting evidence?
- How much will evaluation cost in time and money?
- How will you collect the evidence you need?
- When will you review the training cycle?
Where to Start
This approach to evaluation has been designed in three stages. You may start an evaluation at any stage. The outcome of one stage should inform the next and therefore it is recommended that all stages be completed if the training programme is to be fully evaluated. The person with overall responsibility for the evaluation should ensure that the evaluation outcomes are employed appropriately and reporting and planning targets are established and met.
(You may need to make your own notes at this stage!)
The Model
This evaluation model is an Action Research cycle which continues throughout the life of Diversity and Equality training programme.
The Evaluation Cycle
Training Needs Analysis
(Review)
Training Delivery
(Deliver)
Training Development
(Plan)
Training Needs Analysis - Review
- Evaluates training need. Undertake this stage to answer the following questions.
Why are we doing this training?
What specific things do we want to achieve?
How will we identify success?
Training Development - Plan
- Evaluates training plan. Undertake this stage to answer the following questions.
What are the objectives of the training?
How will we achieve the desired learning outcomes?
Training Delivery - Deliver
- Evaluates training delivery. Undertake this stage to answer the following questions.
What have the trainees learned from the training?
What have we learned about the training?
How will we do things differently in future?
How this Guide is presented
The Model is presented in each of these three stages along with
- a discussion of key issues
- example forms and/or checklist.
There are four appendices.
Appendix 1, Assembling Judging Success and Quality, interpret the evidence and Writing the Report
Appendix 2, Alternative Techniques for collecting evidence,
Appendix 3, Evaluating E-Learning Programmes
Appendix 4 - Emergency Fire Services National Occupational Standards (NOS) Knowledge and Understanding Requirements for Equal Opportunity & Diversity
Review
Training Needs Analysis – Review
Focus questions
Why are we doing this training?
What specific things do we want to achieve?
How will we identify success?
Why Evaluate the Training Need?
The Training Needs Analysis ensures that the right training is being provided to the right people at the right time. There are three aspects to training need
1.The needs of the FRS (Policies and National Standards)
2.The needs of individual (Workplace Training Needs Audit)
3.The needs of the FRS community and the community which the FRS serves(ImpactAssessment).
The FRS needs to provide through training a set of core skills to enable individuals to perform their jobs. Also, individuals will recognise that they need additional skills to do their jobs. Finally, the wider community will recognise skills that FRS members need in order to perform effectively and efficiently.
Policies
Underpinning the FRS core skills are the Policies, (or laws) which the FRS are required to adhere to (as measured by the Equality Standards). It is necessary to ensure that the relevant policies are available, current and applicable. It is not always necessary to convert policies into training content – they do, however, frame the training. Internally written policies often mandate training (sometimes inappropriately). In all cases, policies require interpreting. Amongst the range of possible consequences, training may emerge as one part of the wider Equality Standards implementation plan. It is important to make reasonable judgements about what should/could be included in a training programme. Practical, job related Knowledge, Skills and Attitudes that would best be implemented through training should then be identified. As policies are reviewed, updated or superseded, the consequences on training should also be reviewed.
The use of a simple table may help to keep track of the relevant policies and ensure they are accounted for, available, current and applicable.
(Example)
Policy Audit
Policy / DateEqual Pay Act / 1970
Sex Discrimination Act / 1975
Sex Discrimination (Indirect Discrimination Burden of Proof) Regulations / 2001
Sex Discrimination (Gender Reassignment) Regulations / 1999
Sex Discrimination (Amendment) Regulations / 2003
Race Relations Act / 1976
Race Relations (Amendment) Act / 2000
Race Relations Act (Amendment) Regulations Indirect Discrimination, Burden of Proof, Racial Harassment / 2003
The Criminal Justice and Public Order Act / 1994
Rehabilitation of Offenders Act / 1974
Equal Pay Act / 1970
Equal Value Act / 1983
Disability Discrimination Act / 1995
Protection from Harassment Act / 1997
Employment Directive / 2000
The Human Rights Act / 1998
Employment Equality (Sexual Orientation) Regulations / 2003
Employment Equality (Religion or Belief) Regulations / 2003
National Occupational Standard / 2004
Age Discrimination Act / 2006
Fire and Rescue Service’s Bill - Legal Responsibility / 2004
Health and Safety at Work Act / 1974
Equality Bill – Gender Duty / 2004
(Complete this list with any missing policies and cross out the ones which do not apply)
Once all of the policies have been assembled the following questions should be asked:
(use the space below to note your responses to the questions)
Are all policies listed current, available and applicable?
How should/could content, concepts or information from these polices be incorporated into a training programme?
(e.g. are any of the issues subjects for discussion or testing or should they be provided in text form?)
Are there any implications for training from Equality Standards Reporting?
Role Map & National Standard Audit
Role Maps
A role-map is a bit like a job description. It tells you, in general terms, everything you need to do in your role. It also tells you how well you need to do it. All roles are slightly different (for example a firefighter doesn’t do the same job (role) as a watch manager) but all roles have things in common too. . . . . To prove competent at your job, you will be judged against your role-map. (