--- title: "Postmortem: A Three-Week Tail-Latency Degradation the Dashboard Could Not See" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/incidents/post/p99-tail-latency-three-week-blind-spot --- [Home](/)›[Incidents](/incidents)›[All Categories](/incidents/categories)›[Performance](/incidents/categories/performance) Incidents [Performance](/incidents/categories/performance)[Observability](/incidents/categories/observability) # Three Weeks of "Sometimes Doesn't Load", While p50 Stayed Green [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 14 Oct 2026Updated: 18 Oct 202603 Mins read05 Mins listen [Markdown for AI(opens in a new tab)](/incidents/post/p99-tail-latency-three-week-blind-spot/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR - A support ticket said the dashboard "sometimes doesn't load". Our latency dashboard showed a p50 of 90ms and a green tick, so for about three weeks nobody believed them. - They were right, and the monitoring was lying by omission: p99 sat at 4.2 seconds, and the offending endpoint ran a nine-table join 214 times per request on that account. - The gap between the middle and the tail was the whole story. > A slow response is loud. We had a system where the dashboard could not hear it. ### Three Weeks of "Sometimes Doesn't Load", While p50 Stayed Green Contents [Summary](#summary)[Impact](#impact)[Timeline](#timeline)[Root cause](#root-cause)[Contributing factors](#contributing-factors)[What changed](#what-changed)[Did it hold](#did-it-hold)[What I would do differently](#what-i-would-do-differently) ## [Summary](#summary) The complaint that started this was not about speed. It was a support ticket saying the dashboard “sometimes doesn’t load”, from a customer whose account was roughly forty times larger than our median account. Our latency dashboard showed a p50 of 90ms and a green tick, so for about three weeks nobody believed them. They were right, and our monitoring was lying by omission. ## [Impact](#impact) - A real customer, on one of our largest accounts, intermittently unable to load their dashboard for about three weeks. - The p99 sat at 4.2 seconds against a p50 of 90ms. That 46x spread is not a uniformly slow system; it is a small class of requests doing something completely different from the rest. - On the account that complained, the largest-account p99 reached 9.8 seconds. - Large-account requests thrashed Aurora’s buffer pool, so they slowed everyone else down too. ## [Timeline](#timeline) No dated clock survives. The shape of the three weeks does: 1. A support ticket arrives: the dashboard “sometimes doesn’t load”. 2. The p50 dashboard shows green, and for about three weeks the ticket loses to the graph every time it is looked at. 3. Someone finally slices by percentile and account. The p99 is 4.2s, and on the complaining account the endpoint runs its join 214 times and takes just over four seconds. From our own postmortem, three weeks in: “We spent the first sprint optimising the wrong thing, because the p50 said the system was fine and the p50 is what the dashboard showed by default.” ## [Root cause](#root-cause) The offender was `GET /api/v1/projects`, which built each row from a nine-table join covering the project, its owner, its current status, its two most recent activity entries, and a permissions check. On an account with three projects it ran the join three times and returned in 60ms. On the account that complained it ran 214 times. The join included a correlated subquery against the activity table, which had grown past 400 million rows, so its cost scaled with account age rather than account size. ## [Contributing factors](#contributing-factors) **The dashboard defaulted to p50.** The median of a bimodal distribution describes neither mode. Three weeks of disbelief came from one default. **Cache keys were built wrong before they were built right.** The first caching implementation keyed on the incoming request, including a header the load balancer sets per request. Hit ratio zero, Redis full of single-use entries, and a 4% “improvement” that was noise. It took two weeks and a colleague asking why key cardinality exceeded the user count to catch it. **`KEYS` in production.** The first invalidation implementation used `KEYS project:*`, which blocks Redis while it scans every key. It worked in staging with 400 keys and caused a 900ms stall in production with 2 million. ## [What changed](#what-changed) The full rearchitecture is the case study: CDN and Redis layers in front of the join, with measured before-and-after numbers. The incident-specific changes: 1. **Percentile and account-size slicing in the dashboards**, so a large-account tail cannot hide inside a green median again. 2. **Key cardinality tracked against expected cardinality** as a dashboard panel; it would have caught the cache-key bug on day one. 3. **A hard rule from the incident review**: no change could increase the blast radius of an Aurora failover. 4. **No pattern-based deletion anywhere**, `SCAN`\-based invalidation from a centralised key map instead, because every early staleness bug was a write path that forgot one of the three keys it should have touched. ## [Did it hold](#did-it-hold) Measured four weeks before and after at comparable traffic: p99 4.2s to 1.6s, largest-account p99 9.8s to 2.1s, Aurora read IOPS down 71%. The caveat the review made us write down: caching moved the failure mode, it did not remove it. Two of three post-rollout incidents were correctness bugs, and neither triggered a latency alert, because both were fast. ## [What I would do differently](#what-i-would-do-differently) Put the tail on the dashboard before a customer has to put it in a ticket. The p50 default was not a monitoring decision; it was a default. The full rearchitecture, including the stale-on-error design that turned an Aurora failover into slightly old data, is written up in [A p99 Latency Caching Rearcitecture](/case-studies/post/p99-latency-caching-rearchitecture). Was this useful? ## Tags [#Postmortem](/incidents/tags/postmortem)[#Latency](/incidents/tags/latency)[#Caching](/incidents/tags/caching)[#PostgreSQL](/incidents/tags/postgresql)[#Redis](/incidents/tags/redis)[#Monitoring](/incidents/tags/monitoring) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot&title=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green&summary=A%20largest-account%20customer's%20dashboard%20intermittently%20failed%20to%20load%20for%20about%20three%20weeks%20while%20the%20p50%20dashboard%20showed%20green.%20p99%20was%204.2%20seconds%20against%20a%2090ms%20median%2C%20driven%20by%20a%20nine-table%20join%20running%20214%20times%20per%20request.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green%20https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot&text=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot&title=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot&t=Three%20Weeks%20of%20%22Sometimes%20Doesn't%20Load%22%2C%20While%20p50%20Stayed%20Green "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fp99-tail-latency-three-week-blind-spot&media=&description=A%20largest-account%20customer's%20dashboard%20intermittently%20failed%20to%20load%20for%20about%20three%20weeks%20while%20the%20p50%20dashboard%20showed%20green.%20p99%20was%204.2%20seconds%20against%20a%2090ms%20median%2C%20driven%20by%20a%20nine-table%20join%20running%20214%20times%20per%20request. "Share on Pinterest")[Email]() ## Comments ## Continue on this topic The same subject, covered a different way from the article above. - [ ![](/_astro/hero.DE1RZUf0_DBmNc.webp) Case study ### Cutting p99 Latency 60% with a Caching Rearchitecture Four caching layers across CloudFront, the edge, Redis and a DynamoDB read model, plus the invalidation discipline and stampede protection that made the tail latency stay down instead of moving somewhere else. ](/case-studies/post/p99-latency-caching-rearchitecture) - [ ![](/_astro/hero.CCDF2QlB_Z1iKrg.webp) Article ### Centralized Logging with Loki, Grafana, and Fluent Bit: Making Sense of Your Systems Setting up centralized logging with Loki, Grafana, and Fluent Bit for Kubernetes and microservices: deployment, configuration, and troubleshooting for a reliable observability setup. ](/blog/post/centralized-logging-loki-grafana-fluent-bit) - [ ![](/_astro/hero.Ck2oLF89_Z15tJKl.webp) Code snippet ### Redis Caching Patterns: Cache-Aside, Write-Through & Cache Invalidation Master production-ready Redis caching patterns with practical examples. Learn cache-aside (lazy loading), write-through, consistency patterns, TTL strategies, and cache invalidation techniques to reduce database load and improve application performance. ](/codesnippets/post/redis-caching-patterns-architecture) - [ ![](/_astro/hero.SWD1oABZ_Z1jrGgT.webp) Quiz ### Advanced Frontend Patterns: State Management & Performance Master advanced frontend architecture: state management (Redux, Zustand), performance optimization, code splitting, lazy loading, and responsive design patterns. ](/quizzes/post/advanced-frontend-patterns-quiz) ## You might also enjoy More posts on similar topics ## [Three Dashboards Reported Zero Revenue, and Every Job Exited 0](/incidents/post/cron-silent-zero-revenue-failure) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Data Engineering Summary The failure mode that finally forced the pipeline rewrite was mundane. A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed that file #Postmortem#Cron#SQS+3 tags [read more](/incidents/post/cron-silent-zero-revenue-failure) ## [The Admission Gate Denied an Image Built from a Fork](/incidents/post/slsa-admission-gate-fork-built-image) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Security - CI/CD Summary The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer l #Postmortem#SLSA#Supply Chain+3 tags [read more](/incidents/post/slsa-admission-gate-fork-built-image) ## [A Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Architecture Summary A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region #Postmortem#AWS#Multi Region+3 tags [read more](/incidents/post/single-region-payments-api-regional-outage) 3 related posts