A usage model catches the churn that shows in the logins. The accounts that keep logging in and leave anyway are visible only in what they write to support. On 500 fabricated accounts with 3,777 support tickets, we measured how much a language model reading them adds to the usage model.
The company is invented. Kataja Analytics sells a dashboards-and-reporting platform from Helsinki to mid-market firms, 500 accounts of 10 to 200 seats each. Its churn model runs on usage levels, trends, seat utilisation and support-ticket metadata, and catches the accounts whose logins taper off before the cancellation email arrives.
It did not catch Hedbrunn Systems AB, a 31-seat Swedish account at €12,426 a year. Its usage never moved. What moved was the tone of four support tickets over nine months, the last closing with “no need to treat this as urgent”. The account cancelled in November 2025, and nobody had time to read nine months of ticket history for five hundred accounts.
The account book is synthetic: 500 accounts over 24 months, 3,777 support tickets, and 133 churners, generated from a seeded Python model. Deterministic code controlled every structural signal. A language model wrote only the ticket prose, from briefs that specified the arc and the content but not the words. An answer key records every account’s type and the signals built into its data, kept away from every model.
The accounts divide into types designed to test different failure modes. Some decline visibly in usage and write frustrated tickets; any model catches those. Some, like Hedbrunn, use the product normally and churn anyway, their frustration visible only in the tickets. Some churn for reasons that never touch the product: an acquisition, a budget cut, a champion who leaves. There is no signal to find there. Some complain loudly, get their issues resolved fast, and renew every year. And some show a slow, steady decline in usage and never cancel. The question is which combination of signals catches the most genuine churn with the fewest false alarms.
A language model read each account’s tickets in order and returned a frustration trajectory, an unresolved-issue count, a short narrative and a risk score, without seeing usage data or the churn label. A logistic regression combined that reading with the usage model’s score.
If a customer-success team can review a hundred accounts a month, the usage model alone catches 59.4% of the real churners in that list and the combination catches 66.2%, nine more of them for the same work. Churners that only the text layer puts in its top hundred add up to €1.14M in annual revenue, and the ones that only the usage model ranks that high add up to €716K. Across the whole book, the usage model scores 0.87 AUC (a standard ranking measure, where 1.0 is perfect separation and 0.5 is random), the text layer 0.80, and the two together 0.89.
Each layer has a characteristic miss. The usage model puts half the quietly-unhappy accounts in its top hundred but cannot say why they are there, and misses the other half entirely. The text layer over-reads accounts that complain loudly and renew anyway, but the combination clears nearly all of those false alarms. Thirty accounts churn for reasons outside the product, with no signal built in for any model to find.
The explorer below holds the account book, with both layers’ scores for every account, the breakdowns by account type behind the figures above, and the ticket history each reading produced.
Everything on this page is invented. The data was generated with the answers built in, and nothing in the modelling pipeline could see them. A marking script checked the results against the answer key at the end. The testing method used across the case studies is described on the work page.
The approach transfers to a real customer book. Build a usage model on whatever you already track, add a language model reading the tickets, combine the two, and check which kinds of customer each layer catches and misses.
The numbers will not be the same. They come from the mix of account types we designed, and a real book holds whatever mix it holds. The quietly-unhappy accounts that drove most of the improvement here took four rounds of design. The usage model still puts half of them in its top hundred, and tuning stopped there because making them fully invisible would have meant making them stop being the thing we set out to model. Your equivalent may be easier or harder to find, and the only way to know is to measure on your own accounts.
The data is generated by a seeded Python script: the same seed reproduces the 500 accounts and the answer key byte for byte. The monthly usage table and the ticket briefs are shipped as artefacts that do not reproduce exactly in the current environment, because a redesign of one account type changed the brief count and the generator was not updated to match. The ticket prose, written by a language model from those briefs, is session data and would vary on a re-run. A validator checks the coherence rules before anything else runs. Configuration, generator, validator, feature build, both models, the combination fit and the evaluation run in that order on one seed. Tenure in months is excluded from the trained model, because a fixed observation window turns it into a near-perfect churn proxy.
The code, the dataset, the notebook and the full writeup are in the public repository.
If you sell on subscription and keep a written trail against named accounts, in tickets, CRM notes or account emails, tell us how many accounts you have and how far back the writing goes.