117 of 198 invoices, 59.1%, finished on deterministic parsers the system wrote for itself, one supplier at a time. Writing them cost about what they saved in year one; from year two the same stream runs at an estimated seventh of the all-model cost.
The company is invented. Pyökkipaja Oy is a Lahti contract-furniture manufacturer of about 120 employees. Its large domestic suppliers invoice through structured e-invoicing, straight into the ledger, roughly how 85% of Finnish B2B invoicing works. The rest arrives in the accounts-payable inbox as a PDF and is keyed by hand. Those suppliers are a German fittings supplier, a Swedish fabric house, an Estonian steel shop, five smaller domestic firms, and eighteen one-off suppliers who invoice once or twice a year. That residue is a minority of the invoices and most of the manual effort.
The corpus is a year of that inbox, 198 invoices from 26 suppliers, with an answer key recording the true value of every field before any parsing ran.
An agent reads each PDF as an image and returns a structured record against one shared schema. It is the expensive step, and the only one that can handle an invoice it has not seen before. It read 81 documents over the year, 5,427 fields, with no extraction errors.
An overseer watches those parses supplier by supplier. After six invoices in a consistent layout it writes an ordinary Python parser for that supplier, which goes live only once it reproduces the agent’s accepted output on every invoice seen so far; a router falls back to the agent whenever schema, arithmetic or master data fail. Nine parsers cover eight suppliers. The ninth was learnt in October, after one supplier changed invoicing software on 1 August and its parser refused the new layout rather than guess. Those eight suppliers carry 178 of the 198 invoices; the other eighteen stay on the agent, because automating one invoice a year costs more than it saves.
13,650 fields were marked against the key, and two were wrong. Both come from a packaging invoice printing 50 units at €2,00 where the true order was 2 units at €50,00; the line total is €100,00 either way, so the arithmetic passes and the parser accepts it. That is the one place the corpus was built to lie, and only the key sees it. In production the safeguard is a three-way match against purchase order and goods receipt.
Costs are estimates. The run went through our own subscription, and the figures here come from published prices applied to measured tokens. Year one comes to about $9.05 against $9.61 for reading everything with the agent, the nine parsers having cost roughly what they saved in their first year. Year two needs the agent for 27 documents and costs about $1.31 against the same $9.61, repeating thereafter.
Two deviations stayed in the record. A fuel-surcharge line planted to break a freight supplier’s parser was read as ordinary data instead, because that parser derives its columns from the document instead of assuming a fixed layout; three of the four planted breakers worked. Four two-page invoices fell back unplanned, the parsers for those suppliers having been induced from single-page examples; with three such documents in one layout and one in the other, neither will reach six.
The explorer below holds the year. It opens on an invoice as accounts payable receives it, then the stream month by month, the parsers the overseer wrote, and the marking against the key.
The invoices are synthetic. We generated all 198 and hold an answer key for every field, and neither the agent nor the overseer could read it. Every figure above was scored against that key.
Marking turned up four defects in the key and none in the parsing. Expect the same of whatever you mark a real inbox against.
The mechanism transfers to a real inbox. Its learning threshold, regression gate, fallback on failed validation and marking against something the pipeline cannot see all carry over. The deterministic share depends on your supplier base more than on parser quality, since the ceiling here was set by how many suppliers send more than a handful of invoices a year. Counting that in your own ledger is where an engagement starts.
Six files do the work, in this order: generator, validator, schema, regression gate, router and evaluation script. Each of the nine parsers carries a provenance header naming the six invoices it learnt from and recording that no answer key was consulted while it was written. Every figure on this page comes out of the scoring script.
The public repository holds the code, the corpus, the answer key, the nine parsers and the long-form writeup with the complete account.
If invoices still reach your ledger as PDFs, tell us how many arrive in a month and how many suppliers they come from.