Sixteen tenders written for an invented Finnish maintenance firm, 510 requirements between them, were answered twice, once by 64 attempts from fourteen simulated people with a chatbot, and once by a single workflow run per tender. Twelve have something planted in them that is easy to miss, and the firm should not bid for two of those at all. An answer key written alongside them records the right response to each.
A tender response is a compliance exercise. Find every requirement across the main document, the annexes and the draft contract. Answer each one with evidence. Check that nothing contradicts anything else and that no mandatory form has been missed. A single overlooked line in an annex can disqualify a bid that is otherwise strong.
Both arms read the same documents. The question is whether a workflow with the model inside it, and a check on what comes out, finds more of those requirements than a person asking the same model for help. That is the step our shapes page argues for, and this case measures it.
Visakoivu Oy is an invented 180-person technical and property-services firm in Jyväskylä, bidding for public and private maintenance and installation work. We wrote a facts library for it: certifications with expiry dates, ten reference projects, staff by discipline, insurance cover and three years of financials. Both arms could read that library. Then we wrote sixteen tenders for it to answer, from 14 requirements (a simple maintenance contract) to 54 (a multi-annex public procurement). The 510 requirements across the sixteen were inventoried in an answer key at authoring time, before either arm existed.
Twelve of the sixteen carry traps designed to test different failure modes, fourteen traps in all because two of the larger tenders carry two. Three bury a disqualifying clause deep in an annex. Two set eligibility thresholds the firm genuinely fails, so the correct answer is not to bid. Three contradict themselves: the main document says one thing, an annex says another, and catching it means reading both. Three require a mandatory form mentioned once, in passing, inside an annex, with no clause number of its own. Three set specific format or pricing requirements. The other four are clean.
Two arms answered every tender. Arm A simulates fourteen people, each with an everyday working habit: one is careful and takes no notes, one pastes the whole document in and asks for a complete draft response, one rewrites every paragraph against the tender text, one trusts the buyer’s summary page. Each answered four or five, producing 64 attempts total. Arm B is one integrated workflow, run once per tender with no operator. It extracts the requirements into a matrix, drafts an answer for each row citing the facts library, then runs three deterministic gates. The coverage gate bounces back any draft with an unanswered requirement, unless the draft is already a no-bid whose reason names every unanswered row. The facts gate checks that every specific claim is supported by that row’s own citations. The consistency gate checks that the same fact is not quoted at different values across rows. A router writes the sign-off page. No gate reads the answer key, and one script marked both arms against it.
Arm A produced a distribution. Per-tender medians ran from 64.3% to 93.3% of the requirements covered. The best single attempt reached 100%, on a simple tender with 15 requirements, and the worst 40.6%. Difficulty did not track with size: mean medians were 75% for simple tenders, 73% for medium and 78% for the largest. Where a requirement carried a clause label, people did well: 11 of 12 buried disqualifiers, 9 of 12 format traps. Contradictions were harder, at 4 of 12. On the two tenders Visakoivu cannot win, 2 of 8 attempts declined to bid, both on the same tender. On the three hidden-form traps, the human arm scored 0 of 12.
Arm B produced a floor. The workflow answered every requirement on 11 of the 14 tenders it bid on. Both impossible tenders were refused, and differently. On T11 the workflow drafted all 30 rows and then declined, naming the NMSB certificate Visakoivu does not hold. On T15 it stopped at row 7 of 53, stated the firm’s largest reference project at €1,650,000 against the tender’s €2,000,000 threshold, and listed all 46 remaining rows as deliberately not drafted. No fabricated claim survived to a final response in any of the 16 runs, where 22 of 64 individual attempts carried at least one. The drafting step cites the facts library for every claim, and the cheapest path through the pipeline is to say only what the library supports. The facts gate, which checks each claim against that row’s own citations, passed on all 20 gate reports without bouncing once. The ten bounces that did occur, across four runs, all came from the consistency gate, and all were false positives on correctly cited claims.
Everything both arms produced is in the explorer below. Each of the sixteen tenders is stepped through extract, draft, gates and sign-off with that run’s own artefacts, alongside the 64 individual attempts with their chat transcripts and marked drafts, and the six predictions with their verdicts.
It also lets you answer a tender yourself, which is new here. Pick one of three, read the document, tick the clauses you judge binding and call bid or no-bid; the same coverage logic that marked both arms marks you against the same held-out key, and your dot lands on that tender’s scatter beside the four people and the workflow. Nothing you do there is stored or sent anywhere.
Across all sixteen runs the workflow found 507 of the 510 requirements, 99.4%. The three it missed are T03-R014, T09-R031 and T14-R053, and nothing else was missed anywhere. All three are the same kind: a mandatory form mentioned once, in an annex, with no clause number.
T03 asks for a certificate of paid taxes alongside a business ID and a contact name. T09 buries a pension-insurance certificate the same way. T14’s sits in the enclosure clause of a draft-contract annex, beside two forms that are properly required elsewhere, so it reads as a recap and not as a new obligation.
The human arm scored 0 of 12 on the same class. The workflow’s 0 of 3 was not designed: a model read those documents and built a matrix, and the requirements written to hide from a reader’s attention hid from extraction in the same way. A check that reads documents the same way inherits the same blind spot. Catching this class needs a check of a different kind: reconciling the forms actually attached against every form named in the pack, or scanning annex prose for obligations that carry no clause number.
Everything on this page is invented. We wrote the company, the buyers, the tender documents and the facts library, and the answer key was written at the same time. The work page describes how the testing works across the case studies.
Arm A needs one further disclosure. It is a model of attention, not an observation of real bidders. Which requirements each attempt noticed was decided by a seeded engagement model before the attempts were written. Where the distribution sits on the scale is partly a modelling choice. What the individual attempts show is how a given working habit meets a given document structure.
Two things were not measured. The corpus has no time dimension, so nothing here says whether a person is faster on a simple tender. Prose quality was excluded by design and was not scored. A real tender desk produces outcomes that this corpus does not. Win rates, buyer clarifications and rejected submissions are better evidence than a planted key, and slower to arrive.
The project is published step by step: the seeded corpus generator and its validator, the fourteen habit briefs and the engagement model behind them, the five workflow steps with all three gates, and one marking script that scores all 80 outputs against the key. That script is the only code on either arm’s path that opens the answer key.
Its calibration is published with it, including the three times the marking was wrong before it was trusted. Clause text printed in Finnish inside otherwise English documents drove keyword matching to near-zero on four tenders; drafts written entirely in Finnish were marked low for it, one at 35.3% where re-marking with Finnish stems brought it to 78.4%; and the keyword vat, from VAT, matched inside ordinary Finnish word forms. All three are described with their fixes, and two further limitations are disclosed and left unpatched.
The corpus, both arms’ full output, the marking script and the answer key are in the public repository, together with the long-form writeup this page distils.
If answering tenders eats your best people’s weeks, tell us how many a month you answer and who reads the annexes.