A feedback form at the end of the day is how most companies judge whether a course worked. We ran that form on forty simulated learners whose true learning had been set in advance, beside the before-and-after assessment we built. The form correlated with what people had actually learned at r = 0.10, which is close to no relationship at all.
An attendance list and a satisfaction score are what most training leaves behind. Neither measures what anyone can do afterwards, so a company buying a course cannot tell from the paperwork whether the room learned anything.
The alternative is to test what people can do. Learners answer practical tasks before the programme and different tasks of matched difficulty afterwards, and the shift between the two is the measurement. That shift can only be checked against learning whose true size is already known, which is why the cohort in this case is simulated, with each person’s gain set before anyone wrote an answer.
Kuusiharju Oy is an invented technical-trade company of about 200 people. Forty of its staff sat the two-day AI-Augmented Workflows programme from our training page. Each learner carries a planted improvement, meaning how much they actually gained from the course, recorded in an answer key. How confident each person feels afterwards is set separately from how much they learned, so some people feel they improved when they did not, and some improve without noticing.
Six types make up the forty: eight strong improvers, twelve modest ones, seven confident non-learners who feel the course worked when it did not, seven non-responders, three ceiling cases who start with nowhere to improve, and three fast forgetters who lose most of their gain within two months. The confident non-learners are deliberately the largest problem group, because they are the people a feedback form cannot tell from genuine improvers.
Three instruments measured the same cohort. The assessment, which is what we would run on a real cohort, uses two ten-task forms drawn from the programme’s own skill claims. Half the group sits form A before the course and form B after, the other half the reverse, so nobody sees the same questions twice and an improvement cannot come from familiarity with the paper. The tasks are ordinary work, such as describing a specific check you would build into an AI-assisted process, or saying where in a weekly workflow a validation step sits. The feedback sheet is the industry standard, run here as the comparison. It asks for satisfaction and self-rated improvement, once, after the course. The naive evaluation is how training tends to be measured when someone tries, and it carries three documented bad practices. It takes volunteers only, gives them the same form before and after, and counts people’s own rating of their improvement for half the score.
The feedback sheet correlated with actual learning at r = 0.10, and the satisfaction score did slightly worse at 0.06, both computed against the planted gain for all forty learners. On this cohort, the paperwork a company normally keeps says almost nothing about whether anyone learned. The expectation written before the run was that the figure would come in under 0.30.
The assessment caught six of the seven confident non-learners, using a rule fixed before the run and never adjusted. It compares each person’s rise in confidence with their rise in measured performance, both scaled against the rest of the cohort, and flags anyone whose confidence ran more than one standard deviation ahead of their performance. Eight names came back, six of them the intended ones and two false positives, both genuine modest improvers whose scored gains fell far enough behind their confidence for the rule to fire.
Three of the forty were built to lose most of what they gained within two months, and on the day the course ends the assessment cannot see them. At post they average a measured gain of 0.27 against 0.22 for the twenty genuine improvers, so if anything they look slightly better than the people who kept the skill. By eight weeks the fast forgetters have kept 21.9% of that gain and the genuine improvers 87.8%. Markus Hirvonen’s answers show the decay in plain text. Asked at post for an instruction that returns the same three-line summary every time, he specified the field order, the exact output template and a rule for missing data, then explained that the format and the missing-data rule together are what make the output independent of the email’s wording. Eight weeks later, on the same task, he wrote “Give it a fixed structure – date, item and quantity, then the change, in that order. Keeps the three lines the same each time.” The answer keeps the topic and loses the mechanism that earned the marks. A pre/post pair measures whether the room learned, and a third measurement some weeks out measures whether the workplace kept it. The follow-up a month after the course that our training page promises asks the second question.
Six expectations were written into the design before any data existed. The assessment recovered the overall learning effect, measuring a mean gain of 0.14 against the 0.13 that had been planted. Scores are fractions of the marks available on a form, so a gain of 0.13 means the cohort picked up about a seventh of them. The gap of 0.0100 sits inside the tolerance of 0.0124, and that margin matters in the next section.
Form B came out about seven percentage points harder than Form A despite being matched at design time; because half the cohort started on each form, the cohort figure is protected.
The naive evaluation overstated the cohort’s learning by 41.3%, measuring 0.19 where the truth was 0.13. Removing the repeated form takes away 42.8% of the overstatement and removing the volunteer-only selection 24.1%, but removing the self-report weighting pushes the estimate further from the truth, and nine tenths of the overstatement comes from the three flaws interacting and cannot be attributed to any one of them. An evaluation carrying two or three of these flaws cannot have its bias estimated by adding up what each is worth alone. The full decomposition is in the writeup.
All of the numbers above come from a scorer that reads each answer against the rubric. It agrees with the intended scores on 94.23% of item checks, 9,046 of 9,600. So every verdict was recomputed on the intended scores themselves, which is what a perfectly reliable marker would produce, and two of the six flip, in opposite directions. The recovery of the cohort effect fails under perfect scoring, because the gap grows from 0.0100 to 0.0212 against the same tolerance of 0.0124, and the naive evaluation’s overstatement rises from 41.3% to 57.5%. Under the noisy scorer, the assessment looks more accurate than it is and the naive evaluation looks less distorted than it is. The other four hold: the confident-non-learner rule catches all seven under perfect scoring instead of six, and the feedback sheet cannot move at all because it never reads an answer. Both passes are compared verdict by verdict in the writeup.
Neither pass is the true answer. The scored pass is what a real marking programme produces, the perfect pass is what it is trying to approximate, and the distance between them belongs in any conclusion drawn from the scores.
The explorer below holds the cohort and the marking. All forty learners can be filtered by role and type and opened individually, with their planted gain, the assessment’s estimate and their own answers side by side. The gallery of the seven confident non-learners is the part to read first, because it shows what a satisfaction score would have reported about each of them.
The third view is the robustness flip, which puts the six verdicts as scored against the six under a perfect rater and marks the two rows that change.
Everything on this page is simulated. We wrote the company, the forty learners, their improvements and all 240 responses. A marking script checked the results against the answer key. The work page describes how the testing works across the case studies.
This page tests the ruler, not the training. We have not yet trained a commercial cohort, so nothing here is evidence that our training works, or that any training does. What transfers is the assessment and the way it is run. Forms are piloted before they are relied on, and half the cohort starts on each. Marking uses two raters, or one calibrated rubric with its agreement rate measured and published, and any conclusion about the cohort carries that reliability with it. A third measurement follows some weeks after the course.
The numbers will not be the same. The seven confident non-learners and the three forgetters are there because we put them there, and a real cohort has whatever mix it has. What a real cohort learns is unknown until it is measured.
The files run in the order the work did: the cohort generator and its seeded profiles, both task forms with their rubric, the 240 responses, the scoring layer, and one marking script, the only code that scores against the answer key.
The scorer’s calibration is published with it. It began at 74.80% item-level agreement and reached 94.23% across eight rounds of reading disagreements by hand and broadening patterns only where the detector had demonstrably missed text that matched its brief. In one round, widening a pattern to catch a visible false negative collapsed two items into one trigger, costing about 14 percentage points of agreement on two tasks. Reverting to the strict rule would have lowered the overall agreement, so the widening was kept and the trade-off is documented in the code. A stratified sample of 60 of the 554 residual disagreements is published with the response text, so the classification is checkable.
The forms, the cohort, all 240 responses, the marking script, the scorer calibration and the answer key are in the public repository, together with the long-form writeup this page distils.
If your training evidence is a feedback form, tell us what the programme was meant to change and we will tell you how we would measure whether it did.