Hapax · Case study · House knowledge
Tuomimetsä Oy is an invented retailer with 1,100 staff, 38 stores and a data team that answers about 70 requests a week. The database is real enough to query. The sixty requests were written as they would arrive. The correct answers were locked away before any assistant saw them. Below is what happened.
Each request arrived as one line from a named department. Choose the answer you think the company means, then see what the assistant returned with and without the rules written down.
1 of 5
E-commerce
“How many web orders were placed on Cyber Monday 2025 (1 December 2025)?”
The company does not count cancelled orders as web orders. Without the rules, the assistant included them in all five runs. It noted the 326 figure itself every time but returned 345. When the same question was asked as “web shop orders” instead of “web orders placed,” the assistant answered correctly without any rules: it read “web shop orders” as fulfilled orders and excluded cancellations on its own.
The written rules say: “Cancelled orders are not web sales and are not counted as web orders.”
2 of 5
Marketing and CRM
“How many new members did each store sign up last year? Online sign-ups too please.”
At Tuomimetsä, “last year” is the last completed fiscal year (28 April 2025 to 3 May 2026), not the calendar year. Without the rules, the assistant read it as calendar 2025 in all five runs, and noted the fiscal-year reading as an alternative in all five.
The written rules say: “‘Last year’ is not defined in the metric dictionary. Name the fiscal year you used in the answer.” They then list how past requests were read, by date asked.
3 of 5
Commercial (category management)
“Top 10 products by gross margin in the South region, Q4 of FY24/25.”
Management reporting restates history on today’s regions. In Q4 of FY24/25 the South still included five stores that later became the Capital Region. Without the rules, the assistant used the regions as they stood at the time in all five runs.
The written rules say: “Report history as-is unless the requester asks for as-was, and say which one you used.”
4 of 5
Marketing and CRM
“Hi, how many members were contacted for the Spring Garden Launch 2026 mailing, and how much did those members spend between 4 and 24 May 2026? Thanks”
Campaign spend at Tuomimetsä is counted before returns. Without the rules, the assistant deducted refunds in all five runs.
The written rules say: “net sales are before returns, on lines linked to the member.”
5 of 5
Finance (FP&A)
“Web sales including VAT for Q2 of FY24/25 (periods 4 to 6), one figure per period, before returns please.”
The assistant got the VAT right. Every sales line carries its own rate, and the assistant summed the gross amounts correctly, without applying a flat multiplier. The only thing it got wrong was which date to count the web sales on: the order date instead of the dispatch date. That moved €210 across a period total of €156,427, a difference of 0.13%. In all five runs, the same figure, wrong by the same amount.
The written rules say: “Web sales are recognised on the dispatch date in Helsinki time. On a web line, sale_date is the order date and is not used for web net sales.”
Every question above was answered five times, and the figures shown are from real runs. On the main model, the 14 requests that carry no company rule were answered correctly by every setup, every run.
The same request, answered by the assistant with nothing but the database and by the assistant with the company’s rules written down. The two clean requests are the baseline: both assistants get them right, because no company rule is involved. The other four show where the rules make the difference.
The same sixty-two facts were written two ways: as a single document the assistant reads in full, and as ten small handbooks it opens when it needs them. You might expect ten purpose-built handbooks to outperform a single document. They did not: accuracy was 99.3% with the document and 98.3% with the library, too close to call. Switch between them to see how the same rule reads in each format. Look at how much of the document a single request actually needs.
One store had a week of till receipts posted twice, pushing it from seventh to third in the league table. Nobody had documented the problem. Both assistants spotted the duplicate. Only the one with the written rules ranked the stores the company’s way, per trading day, and got the table right.
Accuracy across all sixty requests, five runs each. Without any written rules, the main model scored 49.7%. With a single written document of the company’s rules, 99.3%.
Every number is scored against a held-out answer key the assistant never saw.
Twelve expectations were written and frozen before any of the 1,800 runs. Six were met and six were not.