Hapax · working demonstration · synthetic corpus, answer key published
A year of PDF invoices arrives from twenty-six suppliers in five languages. An agent reads them; an overseer watches, and writes an ordinary parser for each supplier it comes to know – so the model ends the year handling only what genuinely needs it. Every number below is marked against a held-out answer key.
The company, the suppliers and every invoice below are invented; the pipeline, the parsers and the marking are real. Fields that matter (IBANs, reference numbers, VAT arithmetic) are scored digit for digit.
The left one arrives thirty-two times a year and is worth teaching a parser. The right one arrives once, in a layout nobody will see again – automating it would cost more than it saves. A real accounts-payable stream is mostly heads with a long tail, and the design has to respect both.
Each cell is one document, in stream order. Wine is a model read; indigo is a generated parser; gold marks the documents that fell back after the August redesign or a planted failure. The curve underneath is the running cost of the hybrid pipeline against leaving the model to read everything – both estimates, from measured tokens at published list prices.
Try this – press play and watch the wine cells give way to indigo as each supplier crosses its sixth invoice; scrub to August to catch Työkalu-Tiira’s redesign turning its column gold for six documents, then hover any cell to read that document’s own cumulative cost against the all-model line.
Työkalu-Tiira installed new invoicing software. Its parser failed loudly on the first new-layout invoice – by design, a parser that meets a layout it has not seen refuses rather than guesses – and the documents fell back to the model. Six accepted reads later the overseer had written a second parser, under exactly the rule that produced the first. November and December parsed deterministically again. Self-maintenance, demonstrated.
Try this – open the generated parser and read the first lines: it raises an error the moment a line does not match the layout it learned, which is exactly why the six new-layout invoices fell back to the model instead of being silently misread.
parser_s8v2.py – machine-written, regression-gated against the pipeline’s own accepted reads, never shown the answer key.
Eight suppliers were worth teaching; they carry ninety per cent of the volume. The other eighteen – one or two invoices each – stay with the model on purpose. The ceiling on automation is set by the shape of the supplier base.
| Path | Documents | Fields | Wrong values | Accuracy |
|---|
The two wrong values share one explanation. One November invoice was built to lie at the source – quantity and unit price transposed, the line total identical either way:
Printed: 50 kpl à 2,00 €. Ordered: 2 kpl à 50,00 €. Every automated check passes; the parser reads faithfully; only the answer key (in production, a three-way match against the purchase order) sees it.
Every document was assigned a designed outcome before any parsing ran; the run is scored against that plan, deviations included.
| Class | Designed | Actual | Verdict |
|---|
Your accounts-payable inbox already has this shape – a handful of suppliers sending the same layout every month, and a long tail who don’t. The pattern transfers directly – an agent parses everything at first, an overseer watches which suppliers repeat and writes an ordinary parser for each one it comes to know, and the model’s path shrinks to the one-offs and the odd fallback. Year one roughly breaks even, because the parsers have to earn out their own construction cost before they save anything outright; the saving compounds from year two, once they are written and only the tail is left. The invoices, the answer key and the parsers are published in full – this is one run of a method, and your numbers will differ.
Read the long-form case study · Code and dataset · Contact Hapax