On 240 fabricated month-end reports from a fictional SAP system, with every correct value recorded in advance, a language model parsed each one while a second system learned the repeating formats and wrote ordinary code for them. We measured how much of a twelve-month stream the code handled, and what the run cost compared with putting every document through the model.
The company is invented. Jalavakoski Konepaja Oy is a Tampere machinery manufacturer of about 350 employees whose SAP system was configured in 2003 and left alone since. Month-end arrives as raw text exports from five standard transactions. They have fixed-width columns, repeated page headers, German number formats such as 1.234,56, and totals rows distinguishable from line items only by their indentation. One person has read them competently for two decades, and retires this year.
Rebuilding the ERP is a seven-figure project nobody signs off while nothing is visibly broken. Keeping a language model in the loop means paying per document, with no end date. There is a third option. A language model parses the reports, but a second system watches what the model does and writes ordinary code for the formats that keep repeating. As the code takes over, the model retires itself.
We generated 240 month-end reports in the style of five standard SAP transactions: general-ledger line items, purchase orders by material, sales orders, vendor line items, and warehouse stock levels. The documents have fixed-width columns, German number formats, repeated page headers, and status codes embedded in the data. They look like what a finance team receives from an SAP system configured twenty years ago by a German consultancy. An answer key records the correct value of every field in every document, and no system in the pipeline could read it.
Most of the 240 are routine: the same format repeating month after month, the kind a deterministic parser should handle once it has seen enough examples. Six are variants, a different column layout of an existing report type, appearing late and deliberately too rare for the system to learn from. Fifteen are traps designed to break the generated code: an extra column after a system patch, a customer name containing characters that look like column separators, a currency change mid-report.
And one document carries a silent error. Two columns’ values are swapped while the headers stay in place, so every automated check passes: the schema is correct, the totals reconcile, and the master-data lookups all match. Only the answer key reveals the wrong values.
A language model parsed each document into a structured record. A second system watched those parses accumulate by report type. After eight consistent examples of a type, it wrote an ordinary Python parser for that format. The parser went live only after reproducing the model’s output on every document of its type that had already been parsed, and anything that later failed validation fell back to the model. Five report types produced five parsers. From month three of the twelve-month stream, every routine document parsed deterministically.
184 of the 240 documents, 76.7%, ended up on the deterministic path, against a target of about 75% set before the run. Field accuracy was 99.96% on the model path and 100% on the deterministic path, apart from the one silent-error document.
Of the 15 trap documents, 10 broke the generated parsers as designed and fell back to the model. The other five were currency changes mid-report, and the generated parser handled them correctly, treating currency as data and resolving totals per currency.
The silent-error document parsed cleanly, passed every automated check, and produced 66 wrong field values visible only against the answer key. A line-level cross-check comparing quantity times unit price against the total would catch it, and we recommend running that check on every parse in production.
Costs are estimates, priced against published rates and measured token counts. The hybrid stream cost an estimated $8.51; the all-model counterfactual about $21.81, a saving of roughly 61%. The saving grows with volume, because once a parser exists, a routine document costs nothing.
The model path’s 13 field errors came from a prompt shortened between two parsing waves. The edit dropped a qualifying phrase, and the model captured a whole line where it should have captured the number. Instructions to a model need the same version control as code.
The explorer below shows the whole run: the documents as the finance team receives them, the stream month by month, one of the five parsers the system wrote, and the result for every document checked against the answer key. The five parsers and the field-level marks are in the repository.
Everything on this page is invented. We generated the 240 documents and hold an answer key for every field, which nothing in the pipeline could read. A marking script checked the results against it.
The method transfers to a real ERP. Let a model parse the documents, have a second system learn the repeating formats and write code for them, test each piece of generated code against the model’s own output before it goes live, and fall back to the model on anything that fails. Those steps work on any set of reports. The invoice case runs the same method on PDF invoices, supplier by supplier. The contracts case runs it on real contracts, with someone else’s expert annotations as the answer key.
The numbers will not be the same. No canonical SAP export format exists, which is why we built one. How much of your stream ends up on deterministic code depends on how consistent your own reports are. A short audit of what you actually export settles that, and it is where an engagement starts.
The data is generated by a seeded Python script: the same seed always produces the same 240 documents. A validator checks seven coherence rules before anything runs: arithmetic identities to the cent, cross-transaction consistency, amount plausibility, date integrity. The model-path parses and the generated parsers are recorded artefacts from one-time sessions. The router replays the stream over those artefacts, and the evaluation script scores it against the answer key. The five generated parsers sit in the repository with provenance headers naming the documents each one learned from.
The code, the corpus, the answer key and the full writeup are in the public repository.
If your month-end arrives as text exports that one person can read, tell us which reports they are and how many come through each month.