Summary
A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region to fail over to, so there was nothing to do but wait for the region to come back.
Recovery time was whatever the provider’s recovery time turned out to be. That is the finding.
Impact
Total loss of service for the duration. A payments API fails worse than most things at the same downtime:
- settlements stuck mid-flight rather than merely delayed
- partners escalating against a system that could not answer them
- an availability conversation to have afterwards, because the product is regulated
Timeline
There isn’t one, and I am not going to invent it. The engineering record of the work that followed puts the outage at a few hours but never captured when we detected it or when it cleared.
What is established: the region’s control plane degraded, the service had no second region to move to, and it stayed down until the region recovered. No mitigation existed on our side. A timeline of our actions would be a list of things we could not do.
Root cause
Two separate things, and only one of them was ever ours to fix.
The trigger was a control-plane incident in the provider’s region. Rare, outside our control, and not something an architecture review would have prevented.
The cause was that we ran in one region with no failover target. When the only region is unavailable there is no operational response available. No runbook helps, because every path in it ends at the same unreachable place.
Contributing factors
Years of reliability had been reading as evidence of resilience. The service had been dependable in one region for a long time, and that record was doing the work of an argument that one region was enough. It was never that. It only meant the question had not come up yet.
Write paths were not idempotent. This did not cause the outage, but it shaped what a fix could look like. When a region gets shaky, clients, proxies and queues all start retrying at once. Without idempotency, that retry storm turns one payment into several, so any failover built on those write paths would have swapped an outage for double charges.
No recovery objective was written down. With no agreed RTO or RPO, there was no number a single-region design had to answer to.
What changed
The mandate was to survive the loss of an entire region without losing a committed payment or charging anyone twice. In order:
- Idempotency keys in a globally replicated store, so a retry returns the original result instead of performing the operation again. This came first on purpose. The real problem was never “run in two regions”, it was “make every money-touching operation safe to repeat, then run in two regions.”
- Active-active across two regions rather than a warm standby, so normal traffic exercises the failover path instead of an emergency being the first time anyone tries it.
- Aurora Global Database with sub-second replication and a promotable writer in each region. It fit the EKS and Aurora PostgreSQL the team already ran, and data-residency rules ruled out a single global write master.
- A written RTO and RPO, proven monthly, by deliberately failing a region out under production-like conditions.
On-call needed a failover they could trust at 3am without manual database promotion steps, because manual steps under pressure are how a recoverable incident becomes a data-loss incident. That constraint is why the failover is automatic rather than a runbook.
Did it hold
In the quarter after cutover the primary region degraded twice. Both times traffic moved to the healthy region inside the target window, with no customer-visible errors and no duplicate settlements.
Finance and risk watched one number, the double-charge count, and it stayed at zero. Failures kept happening. Every write path had become safe to repeat.
What I would do differently
Write the recovery objective down before you need it. The missing RTO was not a documentation gap. It was the reason nobody ever had to defend running in one region, and a number on paper forces that conversation while there is still time to have it.
Read a long clean record as the weak evidence it is. Years of uptime in one region tells you nothing about losing that region. It tells you the question has not been asked.
The full build, including the architecture, the idempotent request flow, the failover timeline and what broke during the game-days, is written up in Multi-Region Active-Active for a Payments API.
