4. Testing whether it has landed

A group can give you an 8.7 and three weeks later do exactly the same as before. The reverse happens too: a course that felt uncomfortable because people were challenged on their own behaviour sometimes produces the biggest change. Satisfaction and effect are two different things, measured at two different moments. This chapter is about the second.

Impact is something other than a good feeling

Effect measurement is establishing whether the behaviour from your objectives actually occurs, and whether the problem from your problem statement has become smaller as a result. So you are not measuring the participant's experience, but the outcome of your design. That answer almost never comes in on the last day of the course.

That does not make the ordinary evaluation form worthless. It says something about the room, the pace and your energy, and that is useful for your own adjustments. It simply says nothing about the question you were hired for. Anyone who steers on that form alone optimises their course for pleasantness, and that is a different course from one that changes behaviour.

The evaluation ladder

Evaluation works in four rungs. The higher the rung, the harder it is to measure and the more it tells you.

Step 1 is satisfaction: did the participant find it useful and pleasant? You measure it at the end of the day, in five minutes. Rung 2 is learning: does the participant now know or can they now do something they could not do at the start? Also measurable at the end of the day, with a short test or with an observation in the core exercise. Rung 3 is behaviour: does the participant also do it at work, six to eight weeks later, once the daily routine has returned? Rung 4 is result: has the problem from your problem statement become smaller? You measure that in the organisation's figures, months later, and for that you need a baseline measurement.

Most evaluations get stuck at step 1, because that one is free. The returns start at step 2, and your client's question is only answered at step 3.

A measurement and a moment for each objective

You do not measure your whole programme, you measure your must-have objectives. The objectives from the "would be nice" pile are left out of your measurement, otherwise it becomes a study rather than an evaluation. For each must-have objective you determine the level from chapter 2, and that level points to the measurement. You measure a knowledge objective with a short test at the end. You measure a skills objective with an observation during the core exercise. You only measure a behavioural objective in the workplace, six to eight weeks later.

Then you formulate one concrete question for each objective. Not "what did you think of the course", but "in which situation have you used the three steps, and what happened then?" A question about a situation produces a story with names and context. A question about a judgement produces a figure that gets you nowhere.

What you can do at the end of the day

The last twenty minutes of your programme lend themselves to rungs 1 and 2, and there you can do more than hand out a form. The feedback form asks five to eight questions about content, pace, trainer and applicability; with one open question aimed at behaviour ("which situation are you going to handle differently this week?") you get more out of it. The performance measurement in the core exercise costs you nothing extra, because you are watching anyway: you count how many participants perform the step from your must-goal twice on their own. The verbal round in which everyone says in one sentence what they are taking away, produces words that appear on no form at all, and you hear straight away whether the peg word has stuck. The short knowledge test of five questions works for procedures and regulations, where the answer is either right or wrong. And with the delivered product you assess what the participant hands in rather than their opinion of it.

What you can do after the course

Rungs 3 and 4 call for a second contact moment, and you agree that in the intake meeting. Afterwards you rarely get it thrown in. With a observation in the workplace the manager or a colleague observes three situations and scores on two or three points from your learning objective; agree in advance what they are looking for, otherwise you get back a general impression. On a follow-up afternoon four to eight weeks later everyone brings along one situation in which it worked and one in which it did not; you then measure and reinforce in the same movement. With a round of calls you call four of the twelve participants for ten minutes and ask about a concrete situation: "when did you last do it, and what happened then?" The conversation with the client six to eight weeks later yields what participants do not report themselves, and he knows whether the problem has become smaller. And the figures from the organisation (lead times, numbers of complaints, error reports) are the hardest rung, provided you have that baseline measurement.

Back to the design

In the circular model, evaluating is not an end point but a hinge. What you get back determines where you step back to, and not every signal calls for the same repair. Were participants satisfied but are they doing nothing differently, then your exercise resembled the training room too much and their work too little; you go back to the exploration and collect genuine case material from the target group. Can they do it but are they not allowed to, then the working environment is blocking the behaviour; in that case your problem statement was wrong, because the gap was not with the participant but in the organisation, and you discuss that with the client and not with the group. Did the core exercise overrun and the tail end drop away, then it is a fault in your timing; you adjust the session plan and set a hard time for the start of the core exercise. And one block scored consistently low across three groups, then there is a pattern and you rebuild that block. With one deviating group, you first look at the circumstances of that particular day.

Your own logbook for each training day

Alongside the participants' evaluation you keep a second list: the one about yourself. You fill it in within an hour of finishing, because after that it fades. Three columns: what went well, what needed attention, and which change you are making. That third column is the most important one. A point of attention without an intervention repeats itself with the next group. So do not write down an observation there ("the introduction overran"), but a change to your running order: which part you are shortening, which question you are putting differently, which material you are sending out in advance.

Summary

• Impact measurement looks at whether the objectives from your design have been achieved, not at whether the day went pleasantly. Those two do not run parallel.
• The evaluation ladder has four steps: satisfaction, learning, behaviour and results. Your client's question is only answered at step 3.
• You only measure your must-have objectives. The level of the objective points to the form of measurement and the moment of measurement.
• You measure behaviour six to eight weeks later, once the daily routine has returned.
• At the end of the day: a form with an open question, a performance measurement in the core exercise, a spoken round, a short test or a delivered product.
• After the course: observation, follow-up afternoon, round of calls, a conversation with the client or figures from the organisation. Choose no more than two, and agree them in the intake.
• What you get back takes you back to the exploration or to the problem definition. Which of the two depends on exactly what went wrong.

That completes the circle, and you are back at chapter 1. First the gap, then the programme, and then measuring the gap again.