184 of 240 month-end reports, 76.7%, ended up parsed by ordinary code the system wrote for itself, at an estimated two fifths of the cost of putting every document through a model.
The company is invented. Jalavakoski Konepaja Oy is a Tampere machinery manufacturer of about 350 employees whose SAP system was configured in 2003 and left alone since. Month-end arrives as raw text exports from five standard transactions. They have fixed-width columns, repeated page headers, German number formats such as 1.234,56, and totals rows distinguishable from line items only by their indentation. One person has read them competently for two decades, and retires this year.
Rebuilding the ERP is a seven-figure project nobody signs off while nothing is visibly broken. Leaving a model in the parsing path bills per document with no end date. This case takes the third route.
An agent parses each document into a structured record. An overseer watches those parses accumulate by report type and, after eight consistent examples, writes an ordinary Python parser for that type. It goes live only after reproducing the agent’s earlier output on every document of its type, a check run against the pipeline’s own history and never against the answer key, and anything that later fails validation falls back to the agent. Five report types produced five parsers, and from month 3 of the twelve-month stream every routine document parsed deterministically.
184 of the 240 documents, 76.7%, finished deterministically, against a target of about 75% set before the run. Field accuracy was 99.962% on the model path, 13 wrong fields in 34,470 from a single cause, and 100.000% on the deterministic path excluding the one document carrying a planted silent error, 99.952% including it. Costs are estimates. The model path ran inside our own subscription, so we priced it against published rates and measured token counts, which put the hybrid at about $8.51 and an all-model run at about $21.81, a saving of roughly 61%.
Fifteen documents were built to break the generated parsers. Ten did. Five others carried a currency change mid-report and parsed correctly anyway, because the overseer’s parser resolves totals per currency in general. The generated code passed a test we had written for it to fail, and we reported that rather than design a harder trap afterwards. The 13 model-path errors trace to a brief shortened between two waves of parsing, which lost a qualifying phrase; every one of them passed validation, and briefs are versioned here the way code is.
The explorer below holds the whole run, from the document as the finance team receives it to the stream month by month, the parsers the overseer wrote, and the marking against the key.
The documents are synthetic. We wrote the generator that produced all 240 and hold an answer key for every field, which neither the agent nor the overseer could read. Every figure above is marked against it. The data and privacy page describes the method behind all our case studies.
The method transfers to a real ERP, including the learning threshold, the regression gate, the fallback on failed validation, and the marking against something the pipeline cannot see. The numbers do not. No canonical SAP export format exists to test against, which is why we built one, and your deterministic share depends on how consistent your own reports turn out to be. A short audit of what you actually export settles that, and it is where an engagement starts.
The code runs in order and the files are named to match: configuration, generator, validator, model path, overseer, router, evaluation script, and the five parsers the overseer wrote. Each parser’s header names the documents it learnt from. The figures on this page come from two scripts, one that runs the stream and one that scores it against the answer key.
The code, the corpus, the answer key and the long-form writeup with the complete account are in the public repository.
If your month-end arrives as text exports that one person can read, tell us which reports they are and how many come through each month.