Hapax · working demonstration · synthetic corpus, answer key published
A language model reads a legacy ERP’s spool files; an overseer watches it work, writes ordinary deterministic parsers for the routine formats, and keeps the model only for exceptions. Below, the whole demonstration – run on an engineered corpus with a known answer key, so every number is marked against a known correct answer.
Fixed-width text from a 2003-era installation. Status icons arrive as @0A@, credits carry a trailing minus, one vendor per page, and the totals are labelled in German. One person knew every layout by heart; she retires this year.
Press play. For two months the model parses everything – expensive, slow, correct. After eight consistent documents of a type, the overseer writes that type’s parser; from month three the routine stream is handled by plain code at no marginal cost, and the model is called only when a parser refuses.
Try this – press play, or drag the slider yourself. Watch the tick colour turn from wine (the model, reading the document from scratch) to indigo (the parser it wrote) around month three, and watch the cost line below go flat while the all-model counterfactual keeps climbing.
Cost figures are estimates: published per-token list prices applied to measured subagent token totals (a band, not a point – the central line assumes an 85/15 input/output split). Accuracy figures are exact.
Try this – click any tick to see what happened to that document: which path handled it, what it cost, whether the answer key found anything wrong. Click a gold marker on the cost chart to open the actual parser the overseer wrote at that moment.
Field accuracy is exact, scored against the corpus answer key. The deterministic figure is shown excluding the one document we corrupted on purpose; including it, .
| class | designed outcome | what happened |
|---|
Two of our planted failures were defeated by the system itself: the generated FBL1N parser handled the currency trap we set for it, and an earlier version of the silent error had to be redesigned twice because the parsers kept reading it correctly. Pre-registering the outcomes is what makes that checkable – the deviations are reported as they ran. The one error that survived everything was caught by nothing except the answer key; in production that argument is called sampled reconciliation, and a line-level price × quantity check would have caught this particular one.
None of this is particular to Jalavakoski Konepaja Oy’s five report types. An agent does the manual parsing to start with, at the model’s usual cost per document, while an overseer watches what it produces. Once a report type turns out consistent enough, the overseer writes an ordinary deterministic parser for it, and the model stops being called for anything routine; it stays only for the genuine variants. That pattern transfers to any legacy ERP output old enough that one person still reads it from memory. The cost story is the argument.
Read the long-form case study · Code and dataset · Contact Hapax