A pipeline reads 150 synthetic customer emails to an invented equipment-rental firm, drafts replies from its 30-clause policy handbook, and runs each draft through 3 deterministic gates before deciding whether a person needs to see it. 41 of the emails were written to test the pipeline.
A customer-service inbox receives 150 emails across three weeks. Some are routine: availability checks, booking changes, returns, invoices. Some cite the company’s own policy, sometimes correctly. Some ask for things the policy does not allow. Some concern safety incidents or personal data. Some arrive in threads where a staff member has already made a promise.
The question is whether a pipeline can handle the routine volume automatically while keeping the difficult emails for a person, and whether the gates that enforce that separation actually hold.
Haapaharju Oy is an invented equipment-rental firm in Oulu. We wrote a 30-clause policy handbook for it and generated 150 customer emails with an answer key recording the correct handling of each one. The 41 trap emails are of eight types. Two types, safety incidents and requests for personal data under GDPR, are never answered by the pipeline at all. Discount requests above any staff member’s authority, emails too ambiguous to answer until someone asks a question back, and threads in which a staff member has already promised something belong in the human queue. The remaining three types have a reply the pipeline should send on its own if it drafts it correctly: a customer misquoting the policy is corrected with the real clause, an angry customer who is in the right receives what the policy gives them, and a sympathetic customer whose refund claim the policy denies is refused, citing the clause.
The pipeline runs five stages in order. Classification assigns an intent. Safety and data-request emails shortcut to escalation with no draft generated. Retrieval pulls the full handbook, the thread history and the customer’s account record. Drafting produces a reply. Refusals, corrections and any waiver or discount cite the clause they rest on; other commitments do not. Three deterministic gates then check the draft: a policy gate (is a clause cited where required, does it exist in the handbook, and is any discount within authority?), a consistency gate (does the reply contradict a promise already made in the thread?) and a completeness gate (has every question in the inbound email been addressed?). Routine emails and policy corrections that pass all three go out without human review. Everything else goes to a person.
122 of the 150 emails were auto-sent with zero gate violations, and 6 of them should have been held back for a human. 7 of 7 safety and data-request emails were escalated. The 5 above-authority discount requests and the 6 prior-promise emails were queued, and each prior-promise draft kept the promise it found in the thread. 5 policy misquotes were corrected, citing the real clause, and 4 angry-but-entitled emails were answered on the merits.
The pipeline got 5 of the 8 refund denials right. One of the three misses is a defect in the corpus. The email was written to trigger a deposit forfeiture under a clause that applies to returns more than 7 days late, but the return it describes was 5 days late, and the pipeline read the clause correctly and confirmed that the deposit would be returned.
The failure is on the needs-info emails. All 6 were misclassified as routine and auto-sent. None invented a fact, but all 6 belonged in the human queue, where the ambiguity would have been resolved by asking. The straight-through rate is 81.3%. The 6 inflate it, but without them the rate would still be 77.3%.
The consistency gate never fired. No draft contradicted a promise in the thread history, so the gate was never tested on this corpus. It stays in the architecture because a prior-promise email that gets past the classifier is drafted and sent automatically, and this is the only gate that checks a draft against the thread.
The completeness gate bounced 7 routine emails that should have gone through, and nothing was retuned after the numbers appeared.
The explorer below holds all 150 emails with thread histories, the pipeline’s full decision trace for each one, and the answer key behind a toggle. The flow diagram shows how emails were routed from classification through the gates to the 3 terminal outcomes, with the 7 false blocks as a hatched tributary and the 16 misroutes marked as dots. The scoreboard carries the 6 expectations set before the run and the trap table with per-email detail.
Everything on this page is invented. We wrote the company, the handbook, the customer accounts and all 150 emails. The answer key was written with them and opened by one marking script. The work page describes how the testing works across the case studies.
The architecture transfers: classify, retrieve against a real policy set, draft with clause citations, check against deterministic gates, route with a conservative auto-approved set. Nothing in it knows about equipment rental or Finnish companies.
The numbers will not be the same. The 81.3% straight-through rate, the needs-info failure and the silent consistency gate are properties of this corpus. A different policy handbook, a different model or a different mix of email types would produce different figures.
The corpus, the pipeline outputs, the marking script, the policy handbook and the answer key are in the public repository, together with the long-form writeup this page distils.
If your customer-service inbox runs on templates and judgment calls, tell us what the volume looks like and which policy governs the replies.