--- title: "Postmortem: A Silent Cron Pipeline Failure That Never Alerted" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/incidents/post/cron-silent-zero-revenue-failure --- [Home](/)›[Incidents](/incidents)›[All Categories](/incidents/categories)›[Reliability](/incidents/categories/reliability) Incidents [Next in ReliabilityA Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) [Reliability](/incidents/categories/reliability)[Data Engineering](/incidents/categories/data-engineering) # Three Dashboards Reported Zero Revenue, and Every Job Exited 0 [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 07 Oct 2026Updated: 07 Oct 202603 Mins read05 Mins listen [Markdown for AI(opens in a new tab)](/incidents/post/cron-silent-zero-revenue-failure/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR - A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed it happily, and by morning three internal dashboards confidently reported zero revenue. - Nothing alerted, because nothing had technically failed. Every job exited 0. - The failure that forced the migration to an event-driven pipeline. > Silent failures are worse than loud ones. The loud ones page you. ### Three Dashboards Reported Zero Revenue, and Every Job Exited 0 Contents [Summary](#summary)[Impact](#impact)[Timeline](#timeline)[Root cause](#root-cause)[Contributing factors](#contributing-factors)[What changed](#what-changed)[Did it hold](#did-it-hold)[What I would do differently](#what-i-would-do-differently) ## [Summary](#summary) The failure mode that finally forced the pipeline rewrite was mundane. A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed that file happily, and by morning three internal dashboards were confidently reporting zero revenue. Nothing alerted, because nothing had technically failed. Every job exited 0. ## [Impact](#impact) - Three internal dashboards reporting zero revenue, believed by everyone who opened them, because the number looked like a number. - Nine overnight pages in the eight weeks before this incident class was retired, and four silent failures found only after the fact. - Median time to identify a failing job was about 35 minutes, and that was when someone already knew a job had failed. The first we usually heard of an overnight break was a customer asking why their report was empty. ## [Timeline](#timeline) There is no incident clock. The record that survives is the shape of the night: the zero-row file was produced overnight, consumed by six downstream jobs, and discovered in the morning by people reading dashboards, not by any alert. What is established about the surrounding regime, because it was every night, not one night: - jobs chained by `sleep 900`, scheduled by hope: job A at 02:00, job B at 02:15, because A “usually takes ten minutes”, until A took eighteen and B read a half-written file, which had happened at least twice before - a failed job at 02:14 was simply not done; several were unsafe to run by hand twice, which nobody had documented, so re-running was a small act of faith - output went to `/var/log/cron` on whichever of 6 hosts had drawn the job, rotated weekly, with no record of normal runtimes - of the 41 jobs, we could name a current owner for 12 ## [Root cause](#root-cause) Two causes, one on each side of the interface. The vendor changed a filename without a contract that made that impossible to do quietly. And our jobs treated an empty input as success: a zero-row file was processed completely, exited 0, and passed the emptiness downstream, where six more jobs did the same. ## [Contributing factors](#contributing-factors) **No retries existed.** A failed job was missing until run by hand; the retry story after the migration took a week of chasing duplicate side effects to get right, because retry and idempotency have to be designed together. **Detection depended on customers.** With no alerting, no runtime history, and no owners, the discovery path for a broken overnight job was a customer asking about an empty report. **Forty minutes to find a host.** From the handover: “The thing I remember is spending forty minutes just working out which host a job ran on. Not fixing it. Finding it.” ## [What changed](#what-changed) The full rewrite is the case study: 41 cron jobs across 6 hosts became 3 scheduled jobs over an SQS-driven event pipeline. The pieces this incident directly demanded: 1. **Empty is a failure.** A zero-row input trips an alarm instead of exiting 0. 2. **Retries with idempotency keys**, so a retry is the same work, not new work. The first key design was wrong (it included a timestamp) and produced duplicate PDFs and a very confusing morning; the fixed version later caught a resurrected cron entry on its own. 3. **The alarm moved to DLQ depth**, which cut overnight pages more than any reliability improvement did: 9 in 8 weeks before, 2 in 12 weeks after. ## [Did it hold](#did-it-hold) The two remaining pages were both genuine: a vendor served malformed data for two days, and the DLQ caught every affected message. A commented-out cron entry, uncommented by accident six weeks later, re-ran an already migrated job, and the idempotency layer absorbed the duplicate it was built for. Silent failures after the fact: four before, zero after. ## [What I would do differently](#what-i-would-do-differently) Delete rather than disable. The old cron entries left commented out “just in case” were a live footgun for six weeks, and one of them fired. The full migration, including the visibility-timeout lesson and the measured before-and-after, is written up in [From 41 Cron Jobs to an Event-Driven Pipeline](/case-studies/post/cron-to-event-driven-pipeline). Was this useful? ## Tags [#Postmortem](/incidents/tags/postmortem)[#Cron](/incidents/tags/cron)[#SQS](/incidents/tags/sqs)[#Event Driven](/incidents/tags/event-driven)[#Silent Failure](/incidents/tags/silent-failure)[#Data Pipeline](/incidents/tags/data-pipeline) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure&title=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200&summary=A%20vendor%20changed%20an%20SFTP%20filename.%20One%20cron%20job%20silently%20produced%20a%20zero-row%20file%2C%20six%20downstream%20jobs%20consumed%20it%20happily%2C%20and%20by%20morning%20three%20dashboards%20reported%20zero%20revenue.%20Nothing%20alerted%2C%20because%20nothing%20had%20technically%20failed.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200%20https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure&text=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200 "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure&title=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200 "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure&t=Three%20Dashboards%20Reported%20Zero%20Revenue%2C%20and%20Every%20Job%20Exited%200 "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fcron-silent-zero-revenue-failure&media=&description=A%20vendor%20changed%20an%20SFTP%20filename.%20One%20cron%20job%20silently%20produced%20a%20zero-row%20file%2C%20six%20downstream%20jobs%20consumed%20it%20happily%2C%20and%20by%20morning%20three%20dashboards%20reported%20zero%20revenue.%20Nothing%20alerted%2C%20because%20nothing%20had%20technically%20failed. "Share on Pinterest")[Email]() ## Comments ## Continue on this topic The same subject, covered a different way from the article above. - [ ![](/_astro/hero.Dm63yRWp_Z2rwMA9.webp) Case study ### Replacing Cron Chaos with an Event-Driven Pipeline Forty-one overnight cron jobs with no retries, no ownership and no visibility, moved onto EventBridge, SQS and Step Functions with dead-letter queues and idempotent consumers. ](/case-studies/post/cron-to-event-driven-pipeline) - [ ![](/_astro/hero.C1145KtV_9nTMv.webp) Article ### Designing SLOs and Error Budgets: Your Blueprint for Sustainable Reliability How to design SLOs and error budgets that balance shipping speed with reliability: SLIs, monitoring, business goals, and the policies that turn a budget into a real decision tool. ](/blog/post/designing-slos-error-budgets-reliability-blueprint) - [ ![](/_astro/hero.Db5Lgxho_Z1ojUXw.webp) Dev tip ### AWS VPC Design Best Practices: Subnets, Route Tables, and Security Groups How to lay out an AWS VPC so it survives growth: CIDR planning that leaves room, three subnet tiers, one NAT Gateway per AZ, route tables that define what public actually means, security groups referencing each other instead of CIDRs, and the mistakes that cause production outages. ](/devtips/post/aws-vpc-design-subnets-route-tables-security-groups) - [ ![](/_astro/hero.CiNSoA2B_ZYQDSw.webp) Code snippet ### Python Dataclass Patterns: Slots, Frozen Instances, and Field Validation A Python snippet exploring advanced dataclass patterns for production code. Covers \`\_\_slots\_\_\` with dataclasses, frozen immutable instances, \`\_\_post\_init\_\_\` validation, field factories, and comparison with Pydantic for data modeling. ](/codesnippets/post/python-dataclass-slots-frozen-field-validation) ## You might also enjoy More posts on similar topics ## [A Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Architecture Summary A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region #Postmortem#AWS#Multi Region+3 tags [read more](/incidents/post/single-region-payments-api-regional-outage) ## [The Admission Gate Denied an Image Built from a Fork](/incidents/post/slsa-admission-gate-fork-built-image) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Security - CI/CD Summary The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer l #Postmortem#SLSA#Supply Chain+3 tags [read more](/incidents/post/slsa-admission-gate-fork-built-image) 2 related posts