Why Is Learning Measurement Important?

Why is Learning Measurement important?

Training is an investment to achieve organizational objectives and it is important to determine whether the investment in the training event was worthwhile.

Training costs can be significant in any business. Most employers are prepared to incur these costs because they expect their business to benefit from employees' development and progress.The extent to which businesses have benefited can be assessed by evaluating training appropriately. There are also other good reasons for evaluating training. It helps you to:

track the development of staff knowledge and skills
find out if the learning is being applied in the workplace
identify training gaps and future training needs
establish if the investment was worthwhile
inform future training plans and strategy
ensure training continuously improves

What is good measurement?

Good measurement for learning interventions should include the following:-

Identifying SMART objectivesthat are aligned to the success criteria
Aiming to address all levels of a Learning Measurement Framework
Identifying and gaining agreement on the measurement ahead of developing the learning intervention and selection of the measurement tool in the Measurement Plan
Identifying the right data and tool for measurement

When to take measurement?

Most programs should include data collection immediately following a program. Depending on the measurement objectives, follow-up data collection may be needed. Typical questions that may be answered include:

Immediately after training…

To what extent were the identified training objectives achieved by the program?
What specifically did the learners learn?
What commitment have the learners made about the learning they are going to implement on their return to work?

Follow up…

How successful were the trainees in implementing their action plans?
To what extent were they supported in this by their line managers?

Business Outcomes: ROI (Return on Investment), ROE (Return on Expectations) and Risk Avoidance

There is often the temptation to look at the impact of learning as a financial figure based on the cost of the programme looking at measurable business benefit associated with the cost of that programme. In some cases this can be quantified succinctly (such as learning investment associated with learning a new process or IT system), but longer term ROI may be difficult to prove for behavioral learning programmes. For this reason, it is often worthwhile to express learning measurement in terms of risk reduction or avoidance (the risk of NOT learning towards achieving a strategic objective) or return on expectation (ROE). Given that ROI is difficult to prove through direct links to having learned something (and many CEO’s are realizing that it is not a reliable figure for measuring learning), only consider its use where there is a real and identified need to quantify learning measures in these terms. The investment in the systems and statistical analysis may outweigh the investment costs associated with its calculation when the outcome the organization is seeking is quite obvious and the cost of trying to prove the direct link is not worth the effort.

Benchmarks

Sometime it is appropriate to establish benchmarks. These help to show the improvement in the identified measures. Benchmark data should be collected prior to the training event. There are various ways to set benchmarks:

Existing internal or external data
Estimates of baseline data by participants or others
What is the current state?

Examples of benchmarks are:

Identifying the average score of a Pre-Assessment test
Viewing metrics (sales, performance) for a group of trainees prior to their training event

How to Measure

Defining SMART Learning Objectives

Critical to good measurement is defining the right SMART learning objectives that are aligned to what you are trying to achieve with the learning intervention. Subsequently, all measurement of the impact and effectiveness of a learning intervention should be aligned with the objectives of that intervention. Consider, “What would success look like?” ensure you get the input from the appropriate stakeholders in the business in defining success as this will tie into the measures to demonstrate results (level 4 Level of the Learning Measurement framework.

For example, success would be a reduction in the number of accidents on the manufacturing line. The SMART learning objectives for a training program on health and safety standard operating practice (H&S SOPs) on the manufacturing line may include a range of objectives not just covering demonstration of knowledge of the SOPs, but also applying them. There may also be some behavioral practices that help emphasize how an employee would act in ambiguous situations, such as reporting faults, or choosing the right path to resolve a problem (choices A, B, or C).

The Learning Measurement Framework

Research from empirical data and surveys has shown that 70% of workplace learning is obtained from tough jobs, 20% from people (mainly the boss) and 10% from formal courses or reading (Lombardo & Eichinger. 1996)-

If most learning takes place in the workplace, doing the job, this means that in designing any formal training other actions need to be considered before and after the training. Before training it is important to get clear on the success criteria i.e. change to on the job skills, knowledge, attitude or behaviours, as well as how to measure those criteria after the training. Historically Learning/Training groups have concentrated on measuring the effect of the training itself (known as Level 1 ReactionLevel 2 Learning).

As the 70:20:10 rule is more accepted, it is imperative that all learning interventions/training are designed and measured to deliver impact in the workplace (known as Level 3 Behaviours) as well as meet the success criteria agreed by the stakeholder of the business (know as Level 4: Results).

These four levels of effective learning measurement mentioned above are the KirkpatrickLearning Measurement Framework. This framework is considerd one of the best models in training evaluation and measuring the business value of the leanring programs. It is commonly used framenwork to measure learning. Dr. Don Kirkpatrick created the elegant and simple Kirkpatrick Model in 1954. It is considered a world-wide standard for evaluating the effectiveness of training is commonly .

The most effective evaluation of learning/trainingbegins at the very start of desinging the program.

“Trainers must begin with desired results and then determine what behaviour is needed to accomplish them. Then trainers must determine the attitudes, knowledge, and skills that are necessary to bring about the desired behaviour(s). The final challenge is to present the training program in a way that enables the participants not only to learn what they need to know but also to react favourably to the program.” Donald Kirkpatrick, 1993.

The results are defined in measurable terms, so that all involved can see the ultimate destination of the initiative. Clearly defined results will increase the likelihood that resources will be most effectively and efficiently used to accomplish the mission.

Attempting to apply the four levels after a program has been developed and delivered makes it difficult, if not impossible, to create significant training value. All four levels need to be considered at every step in the program design, execution, and measurement.

Kirkpatrick identified 4 levels of measurement.The 4 levels are described below:

Level / Description and characteristics
1. Reaction / Reaction evaluation is how the delegates felt, and their personal reactions to the training or learning experience, for example:

Did the trainees like and enjoy the training?
Did they consider the training relevant?
Was it a good use of their time?
Did they like the venue, the style, timing, domestics, etc?
Level of participation.
Ease and comfort of experience.
Level of effort required to make the most of the learning.
Perceived practicability and potential for applying the learning.

2. Learning / Learning evaluation is the measurement of the increase in knowledge or intellectual capability from before to after the learning experience:

Did the trainees learn what was intended to be taught?
Did the trainee experience what was intended for them to experience?
What is the extent of advancement or change in the trainees after the training, in the direction or area that was intended?

3. Behavior / Behavior evaluation is the extent to which the trainees applied the learning and changed their behavior, which can be seen/measured immediately following and/or several months after the training, depending on the situation:

Did the trainees put their learning into effect when back on the job?
Were the relevant skills and knowledge used?
Was there noticeable and measurable change in the activity and performance of the trainees when back in their roles?
Was the change in behavior and new level of knowledge sustained?
Would the trainee be able to transfer their learning to another person?
Is the trainee aware of their change in behavior, knowledge, or skill level?

4. Results / Results evaluation is the effect on the business or environment resulting from the improved performance of the trainee
Measures would typically be business or organizational key performance indicators, such as:
Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organizational performance, for instance; numbers of complaints, staff turnover, attrition, failures, waste, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.

It is important that the level of measurement for a program should be linked to the course objectives.

More information can be found on the Kirkpatrick website :

Creating a Measurement Plan

Measurement plans should be created early in the training design process and certainly before developing evaluation tools (e.g. questionnaires etc). This should be your Measurement Plan. This ensures time for any preliminary data collection as well as gaining agreement on the measurement plan before the training event.

If you are clear on the outcome you are aiming for through the learning intervention then early identify the measurement needed to show the extent the objectives and hence outcome has been achieved. This enables you to identify the right tool and craft the right questions based on your planned metrics.

Establishing your Measurement Plan

It is important to document and gain agreement on your measurement strategy prior to data collection. This ensures that you measurement work meets the needs/expectations of the project owners/sponsors. A sample Data Collection plan is displayed below.

Business Outcomes: Measuring Return on Investment, Return on Expecations or Risk Avoidance

Identifying how training provides a good business outcome is always difficult because there are several factors involved. Simply it requires high quality appropriate training, the learner learning and applying on the job, while being enabled to apply their new skills in their business through the support of their management and organisation. Being able to measure this can only be attempted if the business partners are engaged early to help identify what success will look like and to work cooperatively to design the learning intervention aligned to the performance processes..

Before training, learning professionals need to partner with supervisors and managers to prepare participants for training. Even more critical is the role of the supervisor or manager after the training. They are the key people to reinforce newly learned knowledge and skills through support and accountability

Caution: Measuring the impact of learning can become very complex, resulting in expensive surveys, multiple tests, and the use of other assessment tools. Be careful not to make ‘measurement’ THE goal, and weigh the benefits of embarking on complex interventions to prove the learning programmes effectiveness at the cost of good instructional design. That is, if your measurement costs more than the learning programme itself, you have cause for concern and you should rethink the measurement plan to ensure that you are putting the appropriate effort and balance into learning. Think of existing measures and how they are captured and see if you can easily tap into existing means of measuring business outcomes rather than inventing new ones.

1 | Page