#Postmortem
Three Weeks of "Sometimes Doesn't Load", While p50 Stayed Green
- Sev2
- About three weeks
- Performance
- Observability
- 14 Oct 2026
A largest-account customer's dashboard intermittently failed to load for about three weeks while the p50 dashboard showed green. p99 was 4.2 seconds against a 90ms median, driven by a nine-table join running 214 times per request.
Three Dashboards Reported Zero Revenue, and Every Job Exited 0
- Sev2
- Overnight
- Reliability
- Data Engineering
- 07 Oct 2026
A vendor changed an SFTP filename. One cron job silently produced a zero-row file, six downstream jobs consumed it happily, and by morning three dashboards reported zero revenue. Nothing alerted, because nothing had technically failed.
The Admission Gate Denied an Image Built from a Fork
An image built from a fork by a misconfigured pipeline in a sandbox account reached a production cluster and was denied at admission. Under the old setup it would have run, and nobody would have known.
A Regional Control-Plane Failure Took a Payments API Offline
- Sev1
- A few hours
- Reliability
- Architecture
- 08 Sept 2026
A money-movement API ran in one AWS region for years. A regional control-plane incident took it offline for hours with nowhere to fail over to. The fix was idempotency first, then a second region.