Two Ways Our Instrumentation Lied to Us

We run a periodic read-only health sweep across the parts of the product that matter: aggregate metrics, chat, email, search, product usage, revenue. It is deliberately boring. Read production, change nothing, write down what is true.

This week it surfaced two findings that look like opposites. One feature was failing almost every time it ran and our logs had nothing useful to say about it. One experiment was posting a result so strong it would normally end the debate. They turned out to be the same kind of bug, and it is a kind worth naming, because in both cases the thing that was broken was the instrument, not the world it was measuring.

The silence

Our “target this resume to a job” feature takes a job posting, reads it against the user’s resume, and rebuilds the resume for that specific role. It is one of the highest-intent moments we have. Someone has pasted a real job description for a job they actually want.

The sweep pulled the funnel for the week: 22 clicks on the entry point, 22 forms shown, 20 job descriptions submitted, and 3 targeted resumes produced. Seventeen submissions recorded a failure event. Anyone who pasted a posting had roughly a one in seven chance of getting the thing they asked for.

So we went to the failure records, which is what they are for. Every failure event ever recorded, all 34 of them going back a month, was identical:

status: 0
reason: "network_or_parse"
content_type: null

Every single one. Not a mix of statuses with one dominant mode, which would have been a lead. One value, on every row, for a month.

That uniformity is the actual finding. status: 0 is not a status code any server ever returns. It is what our client code stored when a fetch threw before it could read a response, and the surrounding catch was wide enough to swallow a genuinely different set of events: a 500 from the server, a 404 on a moved endpoint, an HTML error page arriving where JSON was expected, a gateway timeout, a request that never left the browser. All of those are distinguishable at the moment they happen. All of them arrived in our database as the same three fields, and by then the difference was gone forever.

A catch-all around a network call is not neutral. It is a decision, usually an accidental one, to destroy the evidence that separates the causes you would act on differently. We had a month of failures, a real user-facing outage inside that month, and a log that could tell us only that something went wrong somewhere, which we already knew from the fact that the resumes were not being produced.

The current code preserves the HTTP status, the content type of what actually came back, and a failure class that distinguishes those cases. That shipped before this sweep ran.

Which brings the second half of the lesson, the part that is easy to skip past. Since that fix deployed, there have been zero failures of any kind, so the new fields have never once been populated in production. We shipped telemetry. We do not yet have telemetry. Those are different states, and the gap between them is only visible if you go and check whether the new path has actually been exercised rather than assuming that merged means working.

There was one more quiet correction in this. Our own handoff notes recorded that the failures had stopped on a particular day. The daily curve showed six more the day after that, and one the day after that. Nobody lied. Someone wrote down a reasonable inference at the time, it hardened into a fact through repetition, and it stayed unchecked until something re-derived it from the data. Notes about production drift out of date in exactly the way production does not.

The result that was too good

The second finding came from the A/B layer. One experiment, testing where a landing page sends people, reported this:

Arm A: 701 impressions, 146 conversions (20.83%)
Arm B: 709 impressions,  57 conversions ( 8.04%)
p = 0.0000

Our standup tool flagged it, correctly by its own rules, as ready to conclude. A 13 point absolute lift on a four figure sample with a vanishing p-value is, on its face, a finished conversation.

We did not conclude it, because that exact shape is a signature we have been burned by before. When one arm of an experiment outperforms the other by an implausible margin and the p-value collapses to zero, the first hypothesis should not be “we found something huge.” It should be “only one of these arms is wired to record conversions.”

Consider what a one-sided wiring bug produces. Both arms serve traffic normally, so impressions look healthy and balanced, which is exactly what makes it convincing. But conversions only fire on one path. The winning arm reports its true rate. The losing arm reports whatever leaks through by other routes. The statistics are then computed faithfully on top of a broken input, and significance math has no way to notice, because a p-value describes how surprising a difference is under an assumption of correct measurement. It cannot see the assumption failing underneath it. The more traffic you send, the more confident the wrong answer becomes.

In this case there was already a note in our state documentation recording that this experiment’s historical assignments were contaminated by a since-fixed wiring problem, and that only data collected after a specific release could support a winner. The tool that computed the significance had no access to that note. It is a good tool. It simply has no notion of provenance, and no automated significance test does. A number can be arithmetically correct and still be an artifact of the pipe it came through.

The companion case in the same report is the same bug wearing the opposite mask. Another experiment, on pricing display, has accumulated 2,178 impressions across its two arms over its lifetime, split almost perfectly evenly, and has recorded zero conversions. Not a low rate. Zero, on both arms, ever.

Zero-ever with real impression volume is not a product finding about how nobody wants the thing. It is a wiring hypothesis, and it should be treated as one until someone traces the conversion event end to end and proves it can fire at all. The rule we keep relearning is that a conversion code has to match at every point it appears: where the experiment is defined, where the impression fires, and everywhere the conversion is supposed to be recorded. Miss one of those and you get a perfectly healthy looking experiment that is structurally incapable of ever reporting success.

What these two have in common

One failure was invisible because the instrument threw away the distinguishing detail and reported a uniform, useless signal. The other was invisible because the instrument reported a signal so strong that it read as a conclusion instead of a symptom. Opposite surfaces, same underlying situation: in both cases the data was telling us about the measurement apparatus, and we were reading it as a statement about users.

That is the hard part of instrumentation. When a sensor breaks, it does not usually go blank in a way that announces itself. It keeps producing well formed, plausible, confidently typed numbers, and those numbers flow into dashboards and significance tests and standup reports that have no way to distinguish a real signal from an artifact of their own plumbing. Absence of a signal, and an implausibly clean signal, are both first and foremost claims about the instrument.

The practical version, which is what we actually changed our habits around:

Takeaways

• Log the discriminator, not just the count. A failure record that cannot separate a 500 from a timeout from a parse error will tell you that something is broken, which you already knew, and nothing else.
• A broad catch around a network call is a decision to discard evidence. Capture the status, the content type, and a failure class at the moment they still exist, because a month later they do not.
• Shipping telemetry is not having telemetry. Confirm the new path has actually been exercised in production before you rely on it during the next incident.
• Zero conversions with real impressions is a wiring hypothesis first and a product finding second. Trace the event end to end before you draw a conclusion about demand.
• An implausibly strong experiment result deserves the same suspicion as an implausibly absent one. Ask whether both arms can record a conversion before you ask what the winner has in common.
• Significance math cannot see provenance. It faithfully computes on whatever it is handed, so the question of whether the input is trustworthy has to be answered somewhere else, by someone who knows what changed and when.
• Notes about production go stale in the same way production does. Re-derive load bearing facts from the data instead of inheriting them from the last person who wrote them down, including yourself.