Sixty requests to a fictional Finnish retailer, each with the correct answer written first and held out. Without written rules, an AI assistant scored 49.7%. With a single document of definitions and worked examples, 99.3%. A cheaper model with the rules scored 29 points above the main model without. Six of twelve predictions were met.
It is Tuesday at Tuomimetsä and a request arrives. Like-for-like growth by region for Q3 last year, please. The analyst knows what the company means by that. The rules are in her head and in a few pinned notes, and nowhere else. Now imagine the company points an AI assistant at the same database. On the like-for-like question, it reads the fiscal calendar, the current regions and the refit closures correctly. Nobody told it which stores are comparable, though. It guessed, and two old tables listing thirty-seven stores agreed. In four of five runs it dropped a store called Tampere Lamminpää, and West came back at two per cent instead of four point eight. Asking twice would give the same answer. With the rules written down in one document, it got the right answer in all five. That question is one of sixty, all built for an invented retailer, Tuomimetsä Oy, with the right answer locked away in advance. Without written rules the assistant scored forty-nine point seven per cent, but with the rules in one document ninety-nine point three, and in ten handbooks ninety-eight point three. The format made no measurable difference. One more finding. A smaller, cheaper model with those handbooks scored seventy-nine per cent, twenty-nine points above the main model without rules. We wrote twelve predictions before the runs and six were wrong. We expected written rules to reach sixty-five to eighty-five per cent, and they reached ninety-eight to ninety-nine. Your company has rules that determine whether a number is right. Write them down, and start with a single document.
If you’d like to see what we could solve for you, write to us at hapax.fi.