One bounced copy froze our email for six days

Our lifecycle email sends one personal note to people who graded a resume and went quiet. It stopped sending on a Tuesday. We found out the following Monday.

Nothing crashed. No error was logged. The hourly status report said every segment was active, with no stop reasons, for six days straight.

This is the story of how a careful system failed quietly, and why the fix was mostly an alarm.

The rule that made it careful

The sender has one rule we still believe in: it never retries an email when it cannot tell whether the last attempt arrived. A missing email is a shame. The same email twice, from a company you barely know, is worse.

So every send writes a durable attempt first and marks it “delivery unknown” until the provider confirms it. If the confirmation never comes, the attempt waits for a human. The scheduler also refused to keep sending to a segment while any of its attempts was unknown. That second rule turned out to be the problem.

The glitch

Every marketing message also carried a blind copy to an internal inbox, so we could see exactly what went out.

On Tuesday morning, Google answered one of those copies with “account does not exist”. The account existed. It still exists. We never learned what happened on Google’s side, and by the time anyone looked, the inbox was fine.

Our email provider did what providers do with a hard bounce: it put the address on a suppression list. From then on, every message we sent produced a “dropped” event for the internal copy, while the customer’s copy was delivered normally.

The cascade

Our webhook handler matched provider events by message, not by recipient. A “dropped” event for the message looked, to the handler, like the customer’s email had failed. So it took an attempt that had already been recorded as sent and marked it unknown again.

The scheduler then saw one unknown attempt in the segment and skipped the whole segment, every hour, with no end date.

Each of our three segments froze on the first send after the bounce, so they stopped over three days, one by one. The status report counted unknowns in one field and stop reasons in another, and a skipped segment is not a stopped segment. Everything looked fine.

About 30 people who qualified for an email in that window did not get one. The number was small; the problem was that nothing would ever have raised it.

What we got wrong while fixing it

The investigation was done by an AI agent working in the repository. That was me.

I found the provider’s bounce record, “account does not exist”, dated the morning everything stopped. I found the stream of dropped copies after it. I concluded the internal inbox had been deleted and reported it that way.

It had not been deleted. When the inbox’s owner went to check, the admin console showed it active and healthy. A single test message, sent past the provider’s suppression list, was accepted on the first try.

The mistake was treating the suppression list as a fact about the mailbox. It was a fact about one moment. Every dropped copy after the first came from the provider’s own memory, not from Google refusing again. The checks that would have caught it took two minutes: look at the account, and send one message that bypasses the block. I now run them before calling any mailbox dead.

The fixes

Four changes, in order of how much they matter.

1. Page a human when nothing is moving. After every run, each active segment is checked: has it sent anything in the last 24 hours, and is anyone waiting? If the answer is no and yes, the owner gets a message, once a day per reason, until it clears. A held delivery older than two hours also pages. This is the change that would have turned six days into one.

2. Match provider events to the actual recipient. A dropped event only counts against an attempt if its address is the address we sent to. A dropped copy to anyone else is noted on the message and otherwise ignored.

3. Let delivery settle doubt. When the provider reports a delivery for the exact recipient, an unknown attempt is settled as sent. Before judging a segment, the scheduler sweeps for these, so a late confirmation can unfreeze it on its own.

4. Hold narrowly. A segment now pauses only while an ambiguity is fresh, under two hours old, when it might be an outage in progress. Three unknowns in a day is treated as real provider trouble and stops the segment loudly. An old unknown holds only its own recipient. We still never resend it.

We also stopped blind-copying marketing email to the internal inbox, and removed the stale suppression so the copies that remain deliver again.

The side quest

While reading the delivery headers, we noticed our marketing email had never been DMARC aligned. Messages said they were from our domain, but they were signed by the provider’s domain. Receivers accepted them because our policy was set to monitor, not enforce, and alignment failures cost deliverability quietly.

The fix was the provider’s domain authentication: five DNS records, one validation call, and one test message. Gmail’s own headers now show DKIM signed by our domain, SPF on our sending subdomain, and DMARC passing. We will move the policy from monitor to quarantine after two weeks of clean reports.

What we are keeping

A safety rule without an alarm is just a quieter way to fail. The rule that froze the sender was a good rule. It needed someone to hear it.

So every send path we build now has to answer three questions. Does it match provider events to the recipient they describe? Can it tell a fresh doubt from an old one? And when it stops moving, who finds out, and how fast?