Forty learners, three instruments.

At Kuusiharju, Kaisa in HR is reading through forty feedback forms from the two-day AI course. The scores average four point two out of five, and that number is all she has when someone asks whether the programme worked. That form and an attendance list are what most training programmes leave behind. Neither measures what anyone can actually do afterwards. This case tests the measurement instrument, not the training. We simulated forty learners and set each person’s true improvement in advance, in an answer key that the instruments score against only at the end. The point is to find out whether it works before we use it on real people.

The cohort includes genuine improvers, people who learnt nothing, and seven people whose confidence rose while their actual skill did not change. Those confident non-learners are the group a feedback form cannot tell apart from people who genuinely improved.

We ran two instruments on the same forty learners. One was the before-and-after assessment we built, which uses practical tasks drawn from the course material. The other was the standard industry feedback form.

The assessment recovered the overall learning to within nought point nought one of the true value. It caught six of the seven confident non-learners. Three people built to forget most of what they learned looked fine on the day, but by eight weeks they had kept only about a fifth of their gain.

The feedback form correlated with what people actually learned to r equals nought point one oh, which is close to no relationship at all. The cohort is entirely simulated. What transfers is the instrument itself.

If you’d like to see what we could solve for you, write to us at hapax.fi.