A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That would have worked, and it would have been the wrong call. Freezing delivery to save money trades a problem you can measure for one you can’t. This is how the bill came down by roughly 41% over two quarters while the team kept shipping through a 25% canary rollout on every release, and why almost none of the win came from turning things off in a panic.
Note
The percentages here are representative of what this pattern achieves, not a single audited client figure. The AWS and Kubernetes mechanics (tagging, Cost Categories, Budgets, Anomaly Detection, Savings Plans, EKS node groups, Karpenter, Argo CD, Argo Rollouts) are exactly as described. Your real savings depend on how much waste you start with and how much of your compute is commitment-eligible.
Impact
The bill came down without a feature freeze. Over two quarters the monthly AWS spend dropped by roughly 41% while the team kept shipping through the same 25% canary it uses for any feature.
None of it came from a panic switch-off. Every lever was reversible and canary-guarded, so the savings held without trading away reliability or delivery speed, and cost turned into a normal signal that shows up in pull requests instead of a quarterly fire drill.
The problem
The bill was growing faster than revenue, which is the signal that matters. Nobody could say where the money went, because nothing was labeled. A single line item for “EC2” across a dozen teams and six node groups tells you nothing you can act on. When finance asked engineering to explain a spike, the honest answer was “we’re not sure,” and that answer is what turns a cost conversation into a feature freeze.
There was also a dashboard, and everyone pointed at it as proof they were “doing FinOps.” A dashboard shows you the number. It does not change anyone’s behavior, and it definitely does not tell an engineer that the node group they oversized last sprint is the reason the graph bent upward. Visibility without attribution is just a prettier version of not knowing.
The last piece was fear. Every proposed saving came with “will this break production?” and without data nobody could answer, so nothing happened. The goal was to make cost a normal, reversible engineering decision instead of a quarterly emergency, and to do it without touching the delivery pipeline the team depended on.
Constraints
A few limits shaped the whole approach.
- No feature freeze. Delivery velocity was the business. Any optimization that slowed shipping was off the table.
- No reliability regressions. Saving money by removing redundancy or headroom was not a real saving.
- Keep the delivery model. Full GitOps, dev to staging to production promotion, and 25% canary rollouts all had to stay exactly as they were.
- Data stays isolated. The data tier runs in subnets with no internet route, and that boundary was non-negotiable.
- Respect confidentiality. Real dollar figures stay private, so success is reported as percentages and unit economics.
Architecture
The SaaS runs entirely on a single EKS cluster per environment, across three availability zones. Each VPC has three tiers of subnets: three public subnets for ingress and NAT, three private subnets for the EKS worker nodes, and three isolated subnets with no internet route for the data tier. Everything the product needs runs in the cluster.
Compute is split into purpose-built managed node groups rather than one big pool, which is what makes both scheduling and cost control tractable. There are separate node groups for the frontend, the backend, data and ETL work, observability, the GitOps controllers, and the internal dashboards, plus a Spot-backed group for batch and preview workloads. Karpenter handles just-in-time node provisioning on top, so capacity follows demand instead of sitting idle.
The whole platform toolchain lives in the cluster, scheduled onto those node groups: Keycloak for single sign-on across the dashboards, Argo CD, and Grafana; SonarQube and Trivy in the delivery path; Argo CD and Argo Rollouts for GitOps and progressive delivery; and Prometheus with Grafana for observability.
Warning
The data tier (RDS with a multi-AZ standby, plus ElastiCache) lives in the isolated subnets. Those subnets have no NAT and no internet gateway route, so the databases cannot reach the internet and the internet cannot reach them. Nodes talk to them only over private VPC routes.
Cost work fails when it lives only in a finance spreadsheet, so the first move was to make spend attributable. A small, enforced tag taxonomy flows into AWS Cost Categories, which maps raw line items to teams and products, and from there into Budgets, Cost Anomaly Detection, and per-team showback.
The taxonomy was deliberately small. Five mandatory tags, not twenty, because a taxonomy nobody follows is worse than none. Environment, CostCenter, Application, and Owner covered almost every question we needed to answer, and ManagedBy flagged anything created by hand instead of through code. On EKS those tags also propagate to node groups and volumes, so cluster compute is attributable per team, not lumped under one anonymous bill.
Enforcement matters more than intent, so the tags were governed centrally. AWS Organizations Tag Policies defined the allowed keys and values, Service Control Policies blocked non-compliant resources, and consolidated billing plus the Cost and Usage Report gave one clean view across every account.
Tip
Chargeback is tempting, but it needs near-perfect tagging and it starts turf wars early. We started with showback: show each team its own spend, let central finance keep paying the bill, and move to chargeback only once the tags were trustworthy. Awareness drove most of the savings before any money changed hands internally.
Delivery: GitOps and progressive rollout
None of the cost work was allowed to disturb delivery, so it helps to see what delivery looks like. Every change runs through the same GitOps pipeline. CI builds and tests, SonarQube enforces a quality gate, and Trivy scans the image and the IaC. Only a clean build pushes to ECR and bumps the image digest in the GitOps manifests repository. Argo CD notices the change and syncs it to the cluster.
Nothing goes fully live at once. Argo Rollouts takes over at the cluster and shifts traffic in steps, starting at 25%, then pausing to check analysis metrics before it widens. If the metrics stay healthy it promotes; if they degrade it rolls back on its own, with no human in the loop. That single behavior is what let the cost changes ship safely, because a right-sized deployment that misbehaved would be caught at 25% of traffic, not 100%.
Environments follow the same path every time. A feature or preview environment spins up per pull request on the Spot-backed node group, merges auto-deploy to dev, the same image digest promotes to staging for integration tests, and only then does it reach production behind the canary.
The canary itself is a few lines of Argo Rollouts config, and it is the same for a feature change or a cost change.
apiVersion: argoproj.io/v1alpha1kind: Rolloutmetadata: name: frontendspec: strategy: canary: steps: - setWeight: 25 - pause: {duration: 10m} - setWeight: 50 - pause: {duration: 10m} - setWeight: 100Implementation
The baseline was code. Rather than tag resources by hand, every provider inherited a default set of tags, so new infrastructure was attributable from the moment it existed.
provider "aws" { region = "us-east-1"
default_tags { tags = { Environment = "Prod" CostCenter = "1001" ManagedBy = "Terraform" } }}Then came the guardrails, automated so nobody had to remember to check a dashboard. A monthly budget with a forecast alert catches planned overspend before the month ends, and Cost Anomaly Detection catches the surprise 3am spike.
aws budgets create-budget \ --account-id 111122223333 \ --budget '{ "BudgetName": "MonthlyCost", "BudgetLimit": { "Amount": "50000", "Unit": "USD" }, "TimeUnit": "MONTHLY", "BudgetType": "COST" }'
aws ce create-anomaly-monitor \ --anomaly-monitor '{ "MonitorName": "CoreServices", "MonitorType": "DIMENSIONAL", "MonitorDimension": "SERVICE" }'With attribution and guardrails in place, the actual optimization ran as normal GitOps changes, each behind the canary, each with a one-commit rollback.
-
Find the waste. Use Cost Explorer grouped by the new tags, plus Kubernetes right-sizing signals from the metrics stack, to rank the most over-provisioned node groups and workloads.
-
Right-size node groups in reversible steps. Drop one instance size or one replica at a time, ship it through the 25% canary, and watch the SLOs. The old manifest is one revert away.
-
Let Karpenter consolidate. Enable consolidation so underused nodes are drained and replaced with fewer, better-packed ones, and move interruptible and preview work to the Spot node group.
-
Put dev and staging to sleep. Scale non-production node groups to zero overnight and on weekends. Nothing runs when nobody is working.
-
Commit last, not first. Only after cluster usage was stable did we buy Compute Savings Plans, so we committed to real baseline usage rather than to waste.
The commitment step is where teams most often lose money, by chasing the deepest discount for a workload they are about to change. The rule we used was simple: match the commitment to the roadmap, not to the current instance.
The team was midway through moving several backend services to Graviton for better price-performance. A three-year EC2 Instance Savings Plan on the old family would have looked cheaper on paper and then stranded the moment those services migrated. A Compute Savings Plan gave up a few points of discount but stayed flexible across families, regions, Fargate, and Lambda, which matters even more on EKS where node groups change shape often. That small premium was cheap insurance.
Results
Over two quarters, the monthly bill came down by roughly 41%, and delivery never paused. Every cost change went out through the same 25% canary as any feature. The breakdown, as representative shares of the total reduction, looked like this.
| Lever | Share of the saving | Nature of the change |
|---|---|---|
| Right-sizing node groups + Karpenter consolidation | Largest | Reversible, canary-guarded |
| Scheduling dev and staging to sleep | Large | Fully reversible |
| Compute Savings Plans matched to the roadmap | Meaningful | 1-year, flexible |
| Spot for batch, preview, and CI | Meaningful | Interruption-tolerant only |
| Storage cleanup and lifecycle policies | Smaller | One-time plus ongoing |
The more durable result was cultural. Cost stopped being a quarterly fire drill. Teams could see their own node-group spend, cost showed up in pull requests as a normal signal, and the “will this break?” fear faded because every change had a canary and a rollback. Treat the percentage as illustrative and the mechanics as the real deliverable.
Lessons
Attribution is the whole game. Nothing else worked until spend had an owner, because you cannot optimize a shared cluster you cannot see per team. The five-tag taxonomy, enforced in code and propagated to node groups, paid for itself before a single node was resized.
Progressive delivery is what makes cost work safe. On a normal deploy model, right-sizing production feels risky enough that teams avoid it. With a 25% canary and automated rollback, a bad resize is a non-event, so the team actually did the work instead of flinching.
Commit to usage, not to hope. The most expensive mistake in cloud cost work is a long, rigid commitment bought early to chase a headline discount. Buy commitments after usage is stable, prefer flexibility while the architecture is still moving, and treat the discount rate as secondary to not stranding the plan.
If I did it again, I would wire a pull-request cost estimate in on day one. Putting the number in front of the engineer at the moment they change a manifest moved behavior more than any dashboard did.
Frequently Asked Questions
A freeze trades a measurable problem for an unmeasurable one. You save some money and lose delivery velocity, customer momentum, and team morale, none of which show up cleanly on the bill. Almost all of the saving here came from waste and mismatched commitments, not from doing less, so the freeze would have hurt the business while barely touching the real cost drivers.
Treat it like any other change. Drop one size or one replica at a time, ship it through the same 25% Argo Rollouts canary as a feature, and watch the SLOs during the pause windows. Let Karpenter consolidate underused nodes rather than doing it by hand. Because every step is a GitOps commit, the rollback is a one-line revert, so a bad resize is caught at 25% of traffic and reverted, not discovered in a postmortem.
Twenty-five percent is a deliberate balance. It is a big enough slice that real traffic patterns and enough metric volume show up quickly, so the analysis step can make an honest call, but small enough that a bad release only touches a quarter of users before it rolls back. Smaller first steps are reasonable for very high-risk changes, but 25% gave this team fast, trustworthy signal without much blast radius.
The isolated subnets have no NAT and no internet gateway route, so the databases cannot reach the internet and vice versa. The application nodes in the private subnets reach RDS and ElastiCache over private VPC routes only. Anything the data tier genuinely needs from an AWS service goes through VPC endpoints, which keep that traffic on the AWS network rather than the public internet.
It makes it safer, which in practice makes it happen. Every cost change is a normal pull request that flows dev to staging to production behind the canary, with SonarQube and Trivy gates on the way. There is no separate risky “cost project,” just ordinary changes with the same guardrails as everything else, so teams approve them quickly.
Enforced tagging. Until spend is attributable per team, product, environment, and node group, every other optimization is guesswork. A small mandatory taxonomy, applied through Terraform default tags and AWS Organizations tag policies, turns the bill from one opaque number into a map you can act on.
