Category
2Incidents
Reliability
Three Dashboards Reported Zero Revenue, and Every Job Exited 0
- Sev2
- Overnight
- Reliability
- Data Engineering
- 07 Oct 2026
A vendor changed an SFTP filename. One cron job silently produced a zero-row file, six downstream jobs consumed it happily, and by morning three dashboards reported zero revenue. Nothing alerted, because nothing had technically failed.
A Regional Control-Plane Failure Took a Payments API Offline
- Sev1
- A few hours
- Reliability
- Architecture
- 08 Sept 2026
A money-movement API ran in one AWS region for years. A regional control-plane incident took it offline for hours with nowhere to fail over to. The fix was idempotency first, then a second region.