--- title: "Multi-Region Active-Active for a Payments API" description: "How a money-movement API was taken active-active across two AWS regions with idempotency keys, conflict-free replication, and a tested RTO and RPO, so a full regional outage never double-charges a customer or loses a committed payment." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/case-studies/post/multi-region-active-active-payments --- # Multi-Region Active-Active for a Payments API A payments API that moves real money had been running comfortably in a single AWS region for years. It was reliable until the day it was not: a regional control-plane incident took the whole service offline for a few hours, and there was no second region to fail over to. For most products that is an outage. For a money-movement API it is stuck settlements, angry partners, and a compliance conversation. The mandate that came out of that incident was simple to say and hard to build: survive the loss of an entire region without losing a committed payment or charging anyone twice. This is the story of taking that API active-active across two regions. The interesting part is not the traffic routing, which is close to a solved problem. The interesting part is the money: making retries safe, keeping two live databases honest, and proving the failover actually works instead of trusting a diagram. ## Impact
Once the second region went live, a full regional failure stopped being an incident and became a drill. During the quarter after cutover the primary region had two brief degradations, and in both cases traffic shifted to the healthy region inside the target window with no customer-visible errors and, most importantly, no duplicate settlements. The number the finance and risk teams cared about was the double-charge count, and it stayed at zero. That is not because failures stopped happening. It is because every write path was made idempotent and every retry, whether from a client, a load balancer, or a queue redelivery, converges on the same result. ## The problem A single-region payments API has two failure modes that a diagram tends to hide. The first is total loss of the region, which is rare but catastrophic and completely outside your control. The second, and the one that actually bites during a failover, is the retry storm: when a region gets shaky, every client, proxy, and queue in the system starts retrying, and if those retries are not idempotent, you turn one payment into several. The business could tolerate a couple of minutes of elevated latency during a failover. It could not tolerate a lost payment that a customer had already seen succeed, and it absolutely could not tolerate charging a card twice. So the real problem was not "run in two regions." It was "make every money-touching operation safe to repeat, then run in two regions." ## Constraints The design had to fit inside some hard limits. Payments are regulated, so data residency rules meant certain records could not leave their region of origin, which ruled out a naive single global write master. The team ran on Kubernetes (EKS) and Aurora PostgreSQL already, so the solution had to build on those rather than introduce an exotic new datastore. And the failover had to be measurable: leadership wanted a specific RTO and RPO written down and proven, not a hand-wave. There was also a people constraint. On-call engineers needed a failover they could trust at 3am without a runbook full of manual database promotion steps, because manual steps under pressure are how a recoverable incident becomes a data-loss incident. ## Architecture Both regions run the full stack and take live traffic. Route 53 uses latency-based routing with health checks so users hit the closest healthy region, and it fails a region out automatically when its health check trips. Each region has its own ALB, API pods on EKS, an idempotency store, and a database. _Active-active across two AWS regions_ Two decisions carry the whole design. The idempotency store is a DynamoDB global table, replicated multi-active across both regions, so a key claimed in one region is visible in the other within about a second. The system of record is Aurora Global Database: a writer in the primary region with sub-second physical replication to the secondary, where a reader can be promoted to writer during a failover in roughly a minute. Committed transactions replicate fast enough that the recovery point stays effectively at zero for anything the customer already saw succeed. The failover path is deliberately boring. A health check trips, DNS shifts, Aurora promotes the secondary, and in-flight retries replay with their idempotency key. _The regional failover timeline (RTO and RPO)_ ## Implementation The idempotency key does the real work here. Every payment request carries an `Idempotency-Key` header. Before doing any work, the API does a conditional write into the idempotency store to claim that key. If the key already exists, the stored result is returned as-is and no charge happens. If the claim succeeds, the API runs the charge inside a database transaction, records the result under the key with a TTL, and returns it. Any retry, from any region, with the same key gets the same answer. _An idempotent charge that survives failover_ The subtle bug to avoid is claiming the key and then crashing before the result is stored, which would leave a claimed-but-unfinished key that blocks the retry forever. The fix is to store an in-progress marker at claim time and let the retry either return the finished result or safely resume, with the transaction as the source of truth for whether the money actually moved. Events flowing out to downstream systems (ledgers, notifications) use a transactional outbox, written in the same transaction as the payment, so an event is emitted exactly once per committed payment and consumers dedupe on the same key. That keeps the two regions from emitting conflicting events for the same operation. ## Results Across the first quarter live, the primary region degraded twice. Both times Route 53 shifted traffic and Aurora promoted the secondary well inside the two-minute RTO target, and customers saw a short latency bump rather than errors. No payment was lost and nothing was charged twice, which was the entire point. The less glamorous result was operational confidence. Because the failover is automatic and every write is idempotent, on-call stopped treating a regional wobble as an emergency. The monthly game-day, where a region is deliberately failed out in production-like conditions, went from a nerve-wracking event to a routine check with a green result. ## Lessons The biggest lesson is that active-active is a data problem wearing a networking costume. Getting traffic to two regions is easy; keeping two live copies of money honest is the hard part, and idempotency is what makes it tractable. If you cannot safely repeat every write, no amount of clever routing will save you during a failover. The second lesson is that an RTO and RPO you have not tested are just wishes. The game-days repeatedly surfaced small issues (a too-aggressive health-check threshold, a client that did not send idempotency keys on one endpoint) that no diagram would have caught. Failover is a feature, and like any feature it has bugs until you exercise it. ## Frequently Asked Questions > **Why active-active instead of active-passive?** Active-passive keeps a warm standby that only takes traffic during a failover, which means the standby path is rarely exercised and tends to rot. Active-active runs real traffic through both regions all the time, so the failover path is the same path you use every day. It costs more, but for a money-movement API the confidence that the second region actually works is worth it. > **How do idempotency keys prevent double charges during a failover?** Every payment request carries a client-generated key. The API claims that key in a globally replicated store before charging, and stores the result against it afterward. If a retry arrives, in the same region or a different one after failover, the key is already present and the original result is returned without charging again. The key, not the region, is what guarantees exactly-once. > **What is the difference between RTO and RPO here?** RTO (recovery time objective) is how long the service can be unavailable before it is back, which here is the couple of minutes it takes DNS to shift and Aurora to promote a writer. RPO (recovery point objective) is how much committed data you can lose, which here is effectively zero because idempotency writes are synchronous and database replication lag stays under a second for committed transactions. > **Does data residency break the active-active model?** It constrains it. Records that legally must stay in their region of origin are not globally writable, so the design keeps the system of record regional (a promotable writer per region) rather than a single global write master. The globally replicated piece is the idempotency store, which holds keys and results, not the regulated ledger data. ## References - [Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) - [Amazon DynamoDB global tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html) - [Making retries safe with idempotent APIs (AWS Builders' Library)](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) - [Amazon Route 53 health checks and DNS failover](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html) - [Transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html)