Assessing the effectiveness of your training – what to evaluate?
“The best training program in the world is absolutely worthless without the will to execute it properly, consistently and with intensity” – John Romaniello
Last week, we looked at the reasons why you should evaluate your online training and learning. Next week, we’ll narrow the focus again and look at online training in detail – how to judge things like presentation, usability and learning impact.
This week, we’ll look at choosing what to evaluate. We’ll focus on Kirkpatrick’s four levels of evaluation, which give a useful framework applicable to all types of training.
Kirkpatrick’s four levels of evaluation
Deciding what to evaluate seems on the face of it simple: you are evaluating the impact of a training or learning programme on the learners.
Now we’ll see why that requires some difficult decisions. At the end of last week’s piece, we looked briefly at Donald Kirkpatrick's summary of reasons to evaluate your training. But in fact, Kirkpatrick is better known for the four levels at which evaluation takes place, which he set out in his book Evaluating Training Programs.
- Reaction – which can be summarised as what the learners themselves thought of the programme
- Learning – how well participants learned new skills or facts
- Behaviour – how participants take their new skills or knowledge and try to apply them to their work
- Results – to what extent the programme yields good results in the form of, for example, improved productivity or better working practices
These are fairly abstract, but they do provide a solid foundation for evaluating training of any kind – online and off. In this article we’ll look at the levels in order, then next week, in part 3, we’ll zoom in on the first two with particular reference to elearning and online training.
Finding out what participants think of a programme is vital: they are best placed to understand their own needs and therefore will have the most accurate sense of how well a programme is meeting those needs – in other words, assessing whether it provides the knowledge they lack, changes their thinking or helps them adopt new skills.
Reaction can be quantitative or qualitative – they’re both useful. An example of quantitative might be scoring a course on a Likert scale, for example. Qualitative reaction might emerge from a focus group or discussion. It is important, however, to have some way of aggregating opinions so that they can be communicated to your human resources department, or to future customers, in an easily-digestible way. We’ll look in detail at evaluation methods next week, but a typical way to gather reaction is to ask participants to complete a survey at the end of a course.
The second level invoves judging how much participants have learned, and is ideally matched to programmes that involved knowledge – the acquisition of facts. The most thorough way to evaluate learning is to pre-assess participants before they begin and assess them after the programme. Kirkpatrick also recommends establishing a control group who do not participate but against whom learners can be compared. Clearly, this level will require more preparation than the first level, but the advantages are clear: evaluating learning is one of the easiest ways to communicate headline results persuasively to stakeholders, future learners, funders and so on. It may well remind you of British school rankings, which are the sum of numerous evaluations bundled into a few simple figures for the benefit of the public.
The aim of learning is “to apply what we learn in different contexts, and to recognize and extend that learning to completely new situations. Collectively, this is called transfer of learning”, according to Robert Haskell in his book Transfer of Learning. Effective training will allow an apprentice mechanical engineer, for example, to design and model a valve in the classroom, then apply that skill and knowledge (ie transfer it) to designing and constructing valves in real life. Not only is transfer of learning hard to achieve, but it is hard to evaluate as it involves some coordination between trainers and those who assess behaviours on the job. Managers who carry out personal development reviews must be aware of what training is intended to achieve and need to be able to feed back on what they see over time.
Kirkpatrick’s fourth level involves looking at outcomes such as return on investment, higher profits, greater workplace engagement, fewer accidents or increased sales. This means making a connection between events that may seem distantly related – between the training programme and results achieved over long periods of time. This is tricky, since other factors may come into play. The team in a distribution warehouse, for example, can improve their efficiency just by doing the job over a long period.
One effective way to evaluate results is to choose a metric such as sales figures, and measure it at regular intervals. In that way, you can begin to build evidence of the impact your programme is having.
Implementing the evaluation levels
Kirkpatrick first published his theories about four levels of evaluation many years ago, but they are still used more or less unchanged in organisations all over the world. This experience suggests that it’s best to implement the levels of evaluation in stages, starting with the easiest – learner reactions – then moving on to learning impact.
Next week, we’ll look at how to apply these first two levels in practical terms to online training and learning.