Three Dashboards Reported Zero Revenue, and Every Job Exited 0

Summary

The failure mode that finally forced the pipeline rewrite was mundane. A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed that file happily, and by morning three internal dashboards were confidently reporting zero revenue.

Nothing alerted, because nothing had technically failed. Every job exited 0.

Impact

• Three internal dashboards reporting zero revenue, believed by everyone who opened them, because the number looked like a number.
• Nine overnight pages in the eight weeks before this incident class was retired, and four silent failures found only after the fact.
• Median time to identify a failing job was about 35 minutes, and that was when someone already knew a job had failed. The first we usually heard of an overnight break was a customer asking why their report was empty.

Timeline

There is no incident clock. The record that survives is the shape of the night: the zero-row file was produced overnight, consumed by six downstream jobs, and discovered in the morning by people reading dashboards, not by any alert.

What is established about the surrounding regime, because it was every night, not one night:

• jobs chained by sleep 900, scheduled by hope: job A at 02:00, job B at 02:15, because A “usually takes ten minutes”, until A took eighteen and B read a half-written file, which had happened at least twice before
• a failed job at 02:14 was simply not done; several were unsafe to run by hand twice, which nobody had documented, so re-running was a small act of faith
• output went to /var/log/cron on whichever of 6 hosts had drawn the job, rotated weekly, with no record of normal runtimes
• of the 41 jobs, we could name a current owner for 12

Root cause

Two causes, one on each side of the interface.

The vendor changed a filename without a contract that made that impossible to do quietly. And our jobs treated an empty input as success: a zero-row file was processed completely, exited 0, and passed the emptiness downstream, where six more jobs did the same.

Contributing factors

No retries existed. A failed job was missing until run by hand; the retry story after the migration took a week of chasing duplicate side effects to get right, because retry and idempotency have to be designed together.

Detection depended on customers. With no alerting, no runtime history, and no owners, the discovery path for a broken overnight job was a customer asking about an empty report.

Forty minutes to find a host. From the handover: “The thing I remember is spending forty minutes just working out which host a job ran on. Not fixing it. Finding it.”

What changed

The full rewrite is the case study: 41 cron jobs across 6 hosts became 3 scheduled jobs over an SQS-driven event pipeline. The pieces this incident directly demanded:

1. Empty is a failure. A zero-row input trips an alarm instead of exiting 0.
2. Retries with idempotency keys, so a retry is the same work, not new work. The first key design was wrong (it included a timestamp) and produced duplicate PDFs and a very confusing morning; the fixed version later caught a resurrected cron entry on its own.
3. The alarm moved to DLQ depth, which cut overnight pages more than any reliability improvement did: 9 in 8 weeks before, 2 in 12 weeks after.

Did it hold

The two remaining pages were both genuine: a vendor served malformed data for two days, and the DLQ caught every affected message. A commented-out cron entry, uncommented by accident six weeks later, re-ran an already migrated job, and the idempotency layer absorbed the duplicate it was built for. Silent failures after the fact: four before, zero after.

What I would do differently

Delete rather than disable. The old cron entries left commented out “just in case” were a live footgun for six weeks, and one of them fired.

The full migration, including the visibility-timeout lesson and the measured before-and-after, is written up in From 41 Cron Jobs to an Event-Driven Pipeline.