The contracts are real. We used 150 SEC filings from a published academic dataset, with expert annotations held out as the answer key. A language model read fifty; a second system wrote pattern-matching code for the categories that kept recurring, and that code then ran alone over the other hundred. Twelve predictions were written before the run. Four were right.
The ERP case and the invoice case use the same method on fabricated data with our own answer key. This case runs it on real contracts, where the answer key is someone else’s work, the predictions went 4 of 12, and about 15 of the 80 disagreements between our pipeline and that key were arguably the key itself.
A due-diligence or legal-operations team asks the same dozen questions of nearly every contract in the stack: who signed, when it starts, whose courts govern it, whether it can be assigned, whether liability is capped, whether competition is restricted. The work is expensive and most of it repeats. The question is how much of it repeats closely enough for ordinary code to take over, and what has to stay with a model that reads.
The contracts are real, which makes this the one study in our portfolio not working from data we generated ourselves. They are 150 commercial contracts filed with the SEC, selected from CUAD, a dataset of 510 contracts annotated by law students and attorneys under The Atticus Project.1 The annotations are the answer key. We did not write it. It stayed out of the pipeline throughout and was read once, after the run had closed.
We selected 12 of CUAD’s 41 annotation categories, spanning the range from near-formulaic language (governing law, document name, agreement date) through partially formulaic (anti-assignment, licence grants, insurance requirements) to genuinely bespoke (IP ownership, non-compete restrictions, caps on liability). 50 contracts went through the learning pipeline. 100 were held out for testing: the matchers’ examination, on contracts no model had read.
Before anything ran, we wrote down twelve predictions about which categories the system would automate. Three we expected to leave the model entirely: governing law, document name, and agreement date. Four we expected to automate partially, with the formulaic instances handled by code and the rest referred back to the model. Five we expected to stay with the model because no amount of examples makes bespoke legal language formulaic.
A language model read 50 contracts, answering every question not yet taken over by a matcher against one schema. A second system watched the answers it accepted and, once a category had accumulated six accepted positives, wrote a deterministic matcher: pattern-matching code with no model behind it. Each matcher had to reproduce the model’s own earlier answers before it went live. Shadow comparisons ran alongside the stream, one per category per batch, 33 in all across 23 contracts. If a shadow and the model disagreed, the matcher was demoted and the category returned to the model.
Twelve matchers were written. One failed its activation gate and never answered a question. Three went live and were demoted, each on a single disagreement: a contract naming Texas, Singapore and Belgium law where the matcher had no jurisdiction outside North America; a preamble scan that overran and took two defined terms for signatories; and a schedule the key itself records under two titles. The eight survivors answered 176 categories across the 100 held-out contracts, 173 of them correctly, 98.3%, at zero marginal cost. A matcher answers only the instances it is confident about and refers the rest back to the model, so that figure is measured over an easier subset by construction.
Four of the twelve predictions were right. None of the three we called fully automatable got there: two were demoted, and the third answered only 21% of the holdout. Four we expected to stay with the model grew matchers that answer a narrow formulaic minority and refer the rest back. The expiration-date matcher was wrong on two of its four holdout answers because an expiration date is computed, not quoted. The audit-rights matcher matched nothing in the holdout because its nine learning contracts phrased the clause differently from all 38 that carry one.
The explorer below holds the run: the twelve categories, the matchers the system wrote, the stream that activated and demoted them, per-category aggregates against the key, and a worked disagreement for each.
The answer key is someone else’s work. CUAD’s annotations were produced by law students and attorneys, valued by the dataset’s authors at over two million dollars at prevailing legal rates. Of 80 disagreements between our pipeline and that key, about 15 were the key in error or arguably so: a misspelled party name, a date in the wrong format for CUAD’s own convention, an agreement date contradicting the span the key itself quotes. The data and privacy page covers how the case studies handle data.
The method transfers to another stack of contracts. Which questions matter and what share of them automates depends on the contracts in front of you, and reading a sample is where an engagement starts.
The steps stand in the order they ran: corpus preparation, answer-key derivation from CUAD’s annotations, a validator that checks the key stayed out of reach, the model path, the code-writing system, the router that replays the stream, and one evaluation script, the only step of the run that reads the answer key, once, after the stream. The twelve matchers are kept as the system wrote them, the failed and demoted ones included, each with a header naming the contracts it learnt from.
The code, the derived data and the full writeup are in the public repository. CUAD’s licence permits us to publish the derived answer-key files,2 and the 150 contracts this case read are there with them. The full CUAD archive is not included in the repository (106 MB); it records where to download it and the checksum to verify it against.
Nothing on this page is legal advice.
Sources
If the same questions get asked of every contract that crosses your desk, tell us which questions they are and roughly how many contracts a month.