A synthetic Finnish purchase ledger is screened three ways – standard audit rule tests, an anomaly-detection model, and three independent AI analysts – and every layer is marked against a held-out answer key. The rules catch everything rule-shaped. The model catches nothing that matters. The analysts triage all of it cleanly, find the one story only reasoning could reach – and still leave the case’s namesake story on the table.
Two automated layers screened the ledger. Seven deterministic rule tests – the audit classics: duplicates, round sums, threshold games, weekend postings, mis-posted accounts – raised 64 flags. An Isolation Forest over eight engineered features raised exactly 200. Every flag was grouped mechanically into 39 verdict units, and three independent analyst runs had to call each one: worry, or stand down, with recomputable evidence either way.
Detector flags cluster where the ledger is busiest, at the year-end seasonal peak. They turn out to be statistical texture.
flags on planted anomalies – 75% first-pass precision, exactly as pre-registered, with 100% recall on every rule-shaped class. The other 16 flags are the designed benign bank: contractual round rent, an agreed stocktake weekend, one coincidental equal-amount pair.
A second pass (recurring-identical suppression, materiality floors) would silence most of the 16. That pass is described in the write-up but was not implemented.
flags on anything planted. The Isolation Forest missed all three designed catches – the drift vendor’s best invoice ranked 818th of 50,000 – and reaching the designed six catches would have required flagging 29% of the whole ledger. A frozen, pre-registered deviation, reported as it ran.
Diagnosis in the notebook: a sustained drift inflates the very baseline it is measured against, and single-invoice vendors present no history to be unusual against.
of 39 unit verdicts correct across three blinded runs. All 26 detector units were correctly stood down by all three – the 200-flag triage burden carried cleanly. Two false positives in total, both the same cautious hedge on one designed benign pair.
364 evidence figures machine-checked: zero disagreements; two −€1,000 headline slips recorded.
Eight anomaly classes were planted, each with a pre-registered prediction of which layer would catch it. Seven behaved. The eighth did not.
| Class | What was planted | Txns | Designed catch | As run |
|---|---|---|---|---|
| D | Duplicate invoices – vendor re-sends after an unanswered week, both get posted | 12 | Rules | Rules – 12/12 flagged |
| R | Round sums and amounts hugging the €10,000 approval tier | 10 | Rules | Rules – 10/10 flagged |
| W | One user posting on weekends and holidays | 8 | Rules | Rules – 8/8 flagged |
| S | Purchases split to stay under the approval tier | 10 | Rules | Rules – 10/10 flagged |
| A | An advertising agency’s invoices mis-coded to IT services | 8 | Rules | Rules – 8/8 flagged |
| V | One supplier as three vendor records – three spellings, one VAT id, three IBANs | 15 | Agents (free hunt) | Agents – found by 3/3 runs, each time as the top action item |
| G | The exemplum: freight silently bundled into goods invoices, then price rises hidden inside the bundle – breadcrumbed in the memos, mirrored in the freight carrier’s decline | 12 | Detector, then agents | Nobody. Detector missed (best rank 818); all three free hunts left it unclaimed |
| C | The ceiling: twelve unremarkable consulting invoices, catchable only with data this ledger does not carry | 12 | Nobody | Nobody – and no run accused it. The trap held both ways |
The case was designed with one class nothing catches. As run, it has two. A ~€1,800-a-month drift on one vendor is invisible in account-level totals. Only a per-vendor movers sweep with a drill-down would have surfaced it, and no run drilled there.
Every unit, its expected verdict derived from the answer key, and what each run actually said. Wine-marked rows are where a run got it wrong.
Try this – open a wine-bordered row below, RU-05 or RU-11 are good ones. Each expands to show what all three runs actually said and the evidence each run read, so you can see a plausible-sounding call go wrong for yourself.
Rule-side units (13)
Detector-side units (26) – every one benign, every one stood down by every run
One German seals supplier existing as three vendor records – Kärrenbach, Kaerrenbach, KARRENBACH – one VAT id, one Hamburg address, three different IBANs, €49,001 of spend split across them. No rule test was built to see it (the byte-exact name check scores zero by design), and it lives across records, not within any one. Every run’s free hunt found it independently; every run made it the top action item; every run named the payment-integrity mechanism.
From July, Teräskontio’s invoices carry "sis. rahtikulut" – freight bundled in, a +8% step – and then price rises hidden inside the bundle, December landing 19% over baseline, while the freight carrier’s own billing shows the mirrored decline. The evidence sits in plain memo text and one vendor-level comparison. The detector missed it; all three analysts, spending their bounded free hunts elsewhere, missed it too. The advisory-page story this case set out to demonstrate is the one no layer found.
And the ceiling class held in both directions: nothing flagged Neuvantila’s twelve unremarkable consulting invoices, and no run accused them without evidence. The production catches this ledger deliberately cannot support – three-way matching, fuzzy vendor-master hygiene, recurring-invoice suppression – are stated in the write-up.
Your own purchase ledger already carries this shape of signal: duplicate submissions, round-sum habits, a vendor whose invoicing has quietly drifted. No extra instrumentation is needed. The method transfers as built. Deterministic rule tests and a statistical detector cover whatever the numbers alone can show, and reasoning agents read memos and vendor records for whatever needs explaining. Every layer is marked against something known before anyone trusts it unattended. On this ledger, the detector caught none of what was planted, and the reasoning layer carried the account.
Read the long-form case study · Code and dataset · Contact Hapax