1,253 Green Tests and Four Production Failures

We shipped a small feature this week: when our resume-advice chatbot writes a rewrite for a user on the free grader page, we show a button, “Open in Builder with these changes,” that seeds an editable document from the user’s uploaded resume and applies the chatbot’s rewrite to it. Straightforward idea, the kind you can describe in one sentence and estimate at half a day.

The unit test suite grew from 1,244 to 1,253 tests while we built it. Every one was green. Then we drove the actual flow on production, repeatedly, using traffic marked with a demo flag that identifies the visitor as test traffic, plus a headless browser clicking the real button after real replies from the real language model. Four separate production-blocking failures showed up, one after another, none of them visible to the unit suite. This is what broke, why the tests couldn’t see it, and what we changed about how we ship.

The setup

The feature has two moments. First, an offer: after the chatbot sends a rewrite in chat, we decide whether to show the “Open in Builder” button under that message. Second, an apply: when the user clicks it, we find the rewrite text in the conversation, open (or reuse) a builder document for that user, write the rewrite into it as a revision, and redirect them into the editor.

Both moments are gated. The offer is gated on “does this look like a rewrite reply, and is this user allowed to use it.” The apply is gated on “can we still find that rewrite, and can we still make the one AI call needed to fold it into the document.” Every gate had unit tests, and every unit test passed. None of that told us whether the feature worked.

Failure one: matching phrasing, not meaning

Our first version decided whether a chat reply “was a rewrite” by checking it against a short list of exact substrings, things like “here’s a revised version.” Clean, fast, and it passed every test we wrote, because we wrote the tests using the same phrases we used to write the check.

On production, the live model said things like “Rewritten:” and “let’s sharpen it. Try:” and “give this a shot instead.” None of those matched, so the button never appeared: the reply text failed a fingerprint test built from our own imagined examples instead of the model’s actual output.

The fix had two parts: widen the substring patterns, and add a second, independent signal that doesn’t depend on exact phrasing at all, namely, did the user explicitly ask for a rewrite, and is the reply long enough to plausibly contain one. A substring list is a snapshot of what we expected a model to say. A live model does not hold still. Any check against generated text has to be built against a distribution of phrasing, not a handful of examples, or it will quietly miss most of what actually gets said.

Failure two: the fixture doesn’t have what production has

Once the button started appearing, it still didn’t show for most real users. The offer gate excluded any session that “already has a builder document attached,” on the theory that if you already have one, we shouldn’t offer to create a second. Reasonable rule, except in production every single grader session already has a builder document quietly attached in the background the moment the session starts. That’s a real performance optimization: we pregenerate the document early so that if the user ever does want to edit, it’s instant. It has nothing to do with whether the user has actually seen or used a builder. But the gate treated “has one attached” as “already used one,” and since production attaches one to everyone, the gate suppressed the feature for essentially all real traffic.

Our test fixtures build a bare session with nothing attached, because that’s the simplest thing to construct, and the simplest thing to construct is rarely the shape production actually has. A fixture encodes what we remembered to set up, not what is true of the live system. This exact trap, treating “has an attribute” as if it means “the user did something,” had already been hit and fixed three separate times elsewhere in the same file, each time with a comment warning the next person. We were the fourth.

Failure three: the free tier’s allowance is zero on purpose

Third failure, once the offer worked correctly: clicking the button did nothing for free users, silently. The apply step checks a generic “can this user make an AI call right now” quota before spending money on the model call that folds the rewrite into the document. That quota check is shared infrastructure, used everywhere we spend money on a model call, and it is also, by design, zero for the free tier, because our paid features are gated behind it on purpose. A feature aimed squarely at converting free users to paid ones was rejected by the exact quota rule built to distinguish free from paid.

Nothing in the unit suite caught this because our fixtures for that endpoint never attached a realistic subscription state. A user object with no subscription record at all behaved differently from a real free-tier user with an exhausted daily allowance, and only the second one exists in production.

Failure four: fixing one side of a gate and leaving the other side stale

Failure one taught us to widen how we detect a rewrite reply, using two signals: pattern match, or (explicit rewrite request and a substantial reply). We fixed the offer step but not, at first, the apply step, which still searched the conversation for the rewrite using only the old strict pattern list. Result: the button now correctly appeared for a wider set of replies, clicked through, and then failed to find the very rewrite it had just offered, because the two halves of the same feature used two different definitions of “this message is a rewrite.”

The fix was straightforward once seen: use the same two-signal check in both places. We kept one deliberate asymmetry, though, and it’s worth stating plainly because it’s a judgment call, not a bug: a long reply only counts as a rewrite if the user actually asked for one. We didn’t relax that. Feeding a long, non-rewrite reply (general career advice, say) into the document-editing call invites the model to manufacture resume content that was never actually written by the user or vetted by us, and our chatbot has a hard rule against inventing facts on someone’s resume. Symmetry between the two gates was the fix; symmetry between “any long reply” and “an actual rewrite” was not, on purpose.

How we found these, and why fast, cheap agents mattered

Each of these four failures was found the same way: an autonomous smoke agent driving the live production site with a headless browser, marked as test traffic via a demo flag so it never touches real user data, clicking through the actual flow end to end and reading the actual response, at a cost of a few cents in model calls per pass. Not a mock, not a fixture: the real button, the real chat reply, the real quota check, the real document write.

Several of these agents worked in parallel over the afternoon, each driving a pass, reporting exactly what broke and where, then handing off a fix to pin down with a regression test before the next pass. That parallelism is what made four rounds of ship, discover it’s broken, fix, prove the fix, ship again fit into an afternoon instead of a week. The tooling is genuinely useful here, but the lesson underneath it isn’t about the tooling, it’s about what kind of testing catches what kind of bug.

Each fix shipped with a new regression test built to reproduce the actual production shape that broke it: a session with a pregenerated document already attached, a user with an exhausted quota, a reply using unmatched phrasing. Those tests didn’t exist before because nobody had reason to imagine those shapes until production showed them to us.

One thing made all four failures fast to diagnose instead of mysterious: the endpoint never fakes success. If the AI call to apply the rewrite fails, or the quota is exhausted, or the rewrite text can’t be found, the user gets a real, specific error, and so did our smoke agent. We could tell what broke from the error text alone, without attaching a debugger to production. An honest-failure contract is worth almost as much as the fix itself, because it makes the next failure diagnosable in minutes instead of hours.

The final pass that afternoon went clean end to end: offer appears, click registers, one AI call folds the rewrite into the document as a revision, redirect fires, and the rewritten bullet is sitting there in the editor, waiting for the user.

Takeaways

• A unit fixture encodes what you remembered to set up about production. It says nothing about what’s actually true of production, and the gap between the two is exactly where these bugs live.
• Anything that checks the output of a live model is checking against a phrasing distribution, not a set of examples. A handful of substrings will always undershoot what the model actually says.
• Features that span multiple gates fail multiplicatively. Fixing one gate and assuming the others still agree with it is its own bug, distinct from whatever the first gate got wrong.
• Make failure honest. If your endpoint reports real, specific errors instead of a generic failure or, worse, a fake success, every future bug gets cheaper to find.
• The loop that actually worked today: ship, drive the real path against production with marked test traffic, fix what breaks, pin it with a regression test built from the real shape, then drive the real path again. Four times, one afternoon, before we trusted it.