The three screening layers from the case study above, scored against a held-out answer key. The catch matrix shows what each layer found and missed. Expand any row to read the evidence each analysis cited.
Two automated layers screened the ledger. Seven deterministic rule tests (the audit classics: duplicates, round sums, threshold games, weekend postings, mis-posted accounts) raised 64 flags. An Isolation Forest over eight engineered features raised exactly 200. The flags were grouped into 39 assessed items, and three independent analyst runs had to call each one worry or stand down, with recomputable evidence either way.
Detector flags cluster where the ledger is busiest, at the year-end seasonal peak. They turn out to be statistical texture.
flags on planted anomalies, for a first-pass precision of 75%, matching the prediction written before the data was screened, with 100% recall on every rule-shaped class. The other 16 flags are the designed benign bank: contractual round rent, an agreed stocktake weekend, one coincidental equal-amount pair.
A second pass (recurring-identical suppression, materiality floors) would silence most of the 16. That pass is described in the write-up but was not implemented.
flags on anything planted. The Isolation Forest missed all three designed catches (the drift vendor’s best invoice ranked 818th of 50,000) and reaching the designed six catches would have required flagging 29% of the whole ledger. The result was committed to in advance and published without adjustment.
Diagnosis in the notebook: a sustained drift inflates the very baseline it is measured against, and single-invoice vendors present no history to be unusual against.
of 39 assessments correct across three independent runs. All 26 detector units were correctly stood down by all three, and the 200-flag triage burden was carried cleanly. Two false positives in total, both the same cautious hedge on one designed benign pair.
364 evidence figures machine-checked: zero disagreements; two −€1,000 headline slips recorded.
Eight anomaly classes were planted, each with a prediction, written before screening, of which layer would catch it. Seven behaved. The eighth did not.
| Class | What was planted | Txns | Rules | Detector | Analysts |
|---|---|---|---|---|---|
| D | Duplicate invoices – vendor re-sends after an unanswered week, both get posted | 12 | designed12/12 | 0/12 | 3/33/33/3RU-02 · RU-03 · RU-04 |
| R | Round sums and amounts hugging the €10,000 approval tier | 10 | designed10/10 | 0/10 | 1/22/22/2RU-05 · RU-06 |
| W | One user posting on weekends and holidays | 8 | designed8/8 | 0/8 | 0/11/11/1RU-13 |
| S | Purchases split to stay under the approval tier | 10 | designed10/10 | 0/10 | 3/30/33/3RU-07 · RU-08 · RU-09 |
| A | An advertising agency’s invoices mis-coded to IT services | 8 | designed8/8 | 0/8 | 1/11/11/1RU-01 |
| V | One supplier as three vendor records – three spellings, one VAT id, three IBANs | 15 | 0/15 | 0/15 | designed1/11/11/1free hunt |
| G | The exemplum: freight silently bundled into goods invoices, then price rises hidden inside the bundle – breadcrumbed in the memos, mirrored in the freight carrier’s decline | 12 | 0/12 | designed0/12 | 0/10/10/1best rank 818 |
| C | The ceiling: twelve unremarkable consulting invoices, catchable only with data this ledger does not carry | 12 | 0/12 | 0/12 | 1/11/11/1 |
| B | Benign exceptions flagged by rules | 16 | designed16/16 | 0/16 | 3/32/32/3RU-10 · RU-11 · RU-12 |
| B | Designed catches the detector missed | 2 | 0/2 | designed0/2ranks 9,069 · 6,641 | – |
Gold marker shows the layer each class was designed for
The case was designed with one class nothing catches. As run, it has two. A ~€1,800-a-month drift on one vendor is invisible in account-level totals. Only a per-vendor movers sweep with a drill-down would have surfaced it, and no run drilled there.
Each group, its expected verdict from the answer key, and what each run actually said. Wine-marked rows are where a run got it wrong.
Try this – open a wine-bordered row below, RU-05 or RU-11 are good ones. Each expands to show what all three runs actually said and the evidence each run read, so you can see a plausible-sounding call go wrong for yourself.
Rule-side units (13)
Detector-side units (26) – every one benign, every one stood down by every run
One German seals supplier existing as three vendor records (Kärrenbach, Kaerrenbach, KARRENBACH), one VAT id, one Hamburg address, three different IBANs, €49,001 of spend split across them. No rule test was built to see it (the byte-exact name check scores zero by design), and it lives across records, not within any one. Every run’s free hunt found it independently, made it the top action item, and named the payment-integrity mechanism.
From July, Teräskontio’s invoices carry "sis. rahtikulut" (freight bundled in, a +8% step) and then price rises hidden inside the bundle, December landing 19% over baseline, while the freight carrier’s own billing shows the mirrored decline. The evidence sits in plain memo text and one vendor-level comparison. The detector missed it. All three analysts, spending their bounded free hunts elsewhere, missed it too.
The ceiling class held in both directions: nothing flagged Neuvantila’s twelve unremarkable consulting invoices, and no run accused them without evidence. The production catches this ledger deliberately cannot support (three-way matching, fuzzy vendor-master hygiene, recurring-invoice suppression) are stated in the write-up.
Your own purchase ledger already carries this shape of signal: duplicate submissions, round-sum habits, a vendor whose invoicing has quietly drifted. Deterministic rule tests and a statistical detector cover whatever the numbers alone can show, and reasoning agents read memos and vendor records for whatever needs explaining. On this ledger, the detector caught none of what was planted, and the reasoning layer carried the account.
Read the long-form case study · Code and dataset · Contact Hapax