Mohammad Abu Mattar | Case Studies

Migrating a Monolith to Kubernetes Without a Big-Bang Cutover

Mohammad Abu Mattar — Mon, 20 Jul 2026 11:54:26 GMT

Almost every failed "let's move off the monolith" project shares one detail: the plan was a big-bang cutover. Rewrite in parallel, pick a weekend, flip the switch, and pray. This is the opposite of that. A large application moved onto EKS one service at a time using the strangler-fig pattern, with a routing facade in front, traffic shifting gradually per route, and a working rollback at every single step. No freeze, no weekend gamble, and no moment where the whole thing was in the air.

Impact

The application reached Kubernetes with no big-bang moment. Every route moved gradually behind a facade with a working rollback, so no single step was ever high stakes and delivery never froze.

The gains were structural rather than a one-time event. Each extracted service got independent deploys and its own scaling, so teams stopped blocking each other and the hot paths no longer forced the whole application to scale with them.

The problem

The monolith itself was not the enemy. It ran fine, the team knew it, and it paid the bills. The problem was that it had become the bottleneck for everything else. Deploys were all-or-nothing, so one risky change held up every other team's work. Scaling meant scaling the entire application even when only one part was hot. And onboarding a new engineer meant handing them the whole thing at once.

The tempting fix, a full rewrite with a cutover, is where teams get hurt. You freeze features to build the replacement, the replacement drifts from the original as the original keeps changing, and the cutover becomes a single high-stakes event with no safe rollback. If anything goes wrong at 2am on migration night, the only option is a panicked revert of everything.

The goal was to get the benefits of independent services without ever betting the business on one cutover. That means the old and new systems have to run side by side, in production, for as long as it takes.

Constraints

No big-bang cutover. At no point could correctness depend on a single switch-flip.
No feature freeze. The monolith kept shipping features throughout the migration.
A rollback at every step. Each increment had to be revertible in minutes, not hours.
No shared-database free-for-all. Extracted services own their data; the goal was decoupling, not a distributed monolith on one schema.
Prove parity before deleting anything. Old code stayed until the new path was verified against it.

Architecture

Before the migration, the shape was familiar: an Application Load Balancer in front of a monolith running across an Auto Scaling group, all talking to one shared relational database.

The target keeps the monolith running, containerized, inside an EKS cluster, and puts a routing facade in front of everything. The facade is the heart of the pattern. It looks at each request and decides whether that path has been migrated to a new service or still belongs to the monolith. Extracted services get their own data stores; the monolith keeps its shared database until its remaining parts are small.

The name comes from the strangler fig, a plant that grows around a tree and gradually replaces it. The new system grows around the monolith, taking over one responsibility at a time, until the original is either gone or small enough to leave alone. Nothing about it requires a dramatic finish.

The routing facade

The facade is where the safety comes from. Every request enters through it, and a route table decides the destination. A path that has been migrated goes to the new service; everything else defaults to the monolith. Crucially, migration of a single route is itself gradual: you shift a small percentage of that route's traffic to the new service, watch it, and widen only when it holds. If the new service misbehaves, the facade falls straight back to the monolith, which is still running and still correct.

In practice the facade can be an ingress with weighted routing, an API gateway, or a service mesh. The mechanism matters less than the property: per-path routing plus per-path traffic weight plus instant fallback.

# /users is being migrated: 10% to the new service, 90% still to the monolith.
http:
  - match:
      - uri:
          prefix: /users
    route:
      - destination: {host: users-service}
        weight: 10
      - destination: {host: monolith}
        weight: 90
  - route: # default: everything else stays on the monolith
      - destination: {host: monolith}
        weight: 100

Implementation

The migration ran as a loop, not a project plan with an end date. Each pass picked one seam, extracted it, shifted traffic, verified, and cleaned up.

Containerize the monolith first. Before extracting anything, get the monolith itself running in EKS behind the facade. Now old and new live in the same place, and the facade is the only thing in front.
Pick a loosely-coupled seam. Choose a capability with a clear boundary and a data set it mostly owns, for example users or billing. Avoid the tangled core on the first pass; early wins build trust.
Build the service with its own data. Give the extracted service its own database rather than pointing it at the monolith's schema. Backfill and keep it in sync during the transition, but the target is independent ownership.
Route to it gradually. Add the path to the facade and shift a small slice of traffic, then widen. Watch latency and error rates during each step, and keep the monolith path warm as a fallback.
Verify parity, then delete. Once the new service matches the monolith's behavior under real traffic, remove that code from the monolith. Deleting the old path is what makes the win permanent.
Repeat, and know when to stop. Move to the next seam. Stop when what remains is small and stable enough that extracting it would cost more than it returns.

The most dangerous shortcut is pointing a new service at the monolith's database so you can "extract later." That gives you two services coupled through one schema, which is a distributed monolith: all of the network overhead, none of the independence. Give the service its own data, even if that means a sync period during the transition.

Data is the genuinely hard part, and it is worth being honest about that. Moving stateless request handling is straightforward; moving the data it owns without downtime is not. The workable approach is to give the new service its own store, backfill it, keep it in sync while both paths run, and cut the monolith's write path over only once the new service is authoritative and verified. Where strict consistency is required during the overlap, treat the monolith as the source of truth until the very last step.

Measure the migration by how much of the monolith is gone, not by how many services exist. A useful signal is the share of production traffic served by extracted services and the amount of code deleted from the monolith. Services created without code removed from the original is motion without progress.

Results

The application moved onto EKS without a single cutover event and without a feature freeze. Because each route shifted gradually with a live fallback, no migration step was a high-stakes moment; the riskiest change only ever touched a small slice of one path at a time. Independent deploys arrived for each extracted service, so teams stopped blocking each other, and the hot paths could scale on their own instead of forcing the whole application to scale with them.

The migration also did not finish in the storybook sense, and that was the right outcome. A stable, low-change remainder of the monolith stayed in place, containerized and behind the facade, because extracting it would have cost more than it returned. Treat "the monolith is gone" as a possible ending, not the goal.

Lessons

The facade is the whole safety story. Because every request always had a valid destination and an instant fallback, no step was irreversible. That single property is what let the team move quickly instead of cautiously.

Extract the easy seams first. The instinct to start with the messy core is a trap. Early, low-risk extractions build the tooling and the team's confidence, so the hard ones later are routine instead of terrifying.

Data ownership is the real migration. The service boundary is easy; the data boundary is the work. Any plan that hand-waves the database is a plan to build a distributed monolith.

Give yourself permission to stop. The goal was never zero monolith. It was independent, deployable, scalable services for the parts that needed it, and a small stable remainder for the parts that did not.

Frequently Asked Questions

It is an incremental migration approach where a new system grows around an old one and takes over its responsibilities one at a time, until the old system is replaced or reduced to a small remainder. A routing facade sits in front and directs each request to either the new component or the old one, so both run in production together and you never need a single cutover.

Because a cutover is a single high-stakes event with no safe rollback. You freeze features to build the replacement, it drifts from the original as the original keeps changing, and if anything breaks on migration night your only option is reverting everything at once. Strangler-fig keeps the old system live the whole time, so every step is small and reversible.

Low coupling and a clear data owner. Pick a capability with a clean boundary that mostly owns its own data, like users or billing, so you are not untangling shared state on your first attempt. Early, low-risk wins build the tooling and the confidence you will need for the harder seams later.

Give each extracted service its own store rather than pointing it at the monolith's schema. Backfill it and keep it in sync while both paths run, then cut the monolith's write path over only once the new service is authoritative and verified. Sharing one database across services is a distributed monolith and defeats the point of the migration.

When the remaining monolith is small and stable enough that extracting more would cost more than it returns. Track the share of production traffic served by extracted services and the amount of code deleted from the monolith. Done does not have to mean zero monolith; a low-change remainder behind the facade is a perfectly good ending.

References

Cutting a SaaS AWS Bill 41% Without Slowing Delivery

Mohammad Abu Mattar — Sun, 19 Jul 2026 12:29:14 GMT

A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That would have worked, and it would have been the wrong call. Freezing delivery to save money trades a problem you can measure for one you can't. This is how the bill came down by roughly 41% over two quarters while the team kept shipping through a 25% canary rollout on every release, and why almost none of the win came from turning things off in a panic.

The percentages here are representative of what this pattern achieves, not a single audited client figure. The AWS and Kubernetes mechanics (tagging, Cost Categories, Budgets, Anomaly Detection, Savings Plans, EKS node groups, Karpenter, Argo CD, Argo Rollouts) are exactly as described. Your real savings depend on how much waste you start with and how much of your compute is commitment-eligible.

Impact

The bill came down without a feature freeze. Over two quarters the monthly AWS spend dropped by roughly 41% while the team kept shipping through the same 25% canary it uses for any feature.

None of it came from a panic switch-off. Every lever was reversible and canary-guarded, so the savings held without trading away reliability or delivery speed, and cost turned into a normal signal that shows up in pull requests instead of a quarterly fire drill.

The problem

The bill was growing faster than revenue, which is the signal that matters. Nobody could say where the money went, because nothing was labeled. A single line item for "EC2" across a dozen teams and six node groups tells you nothing you can act on. When finance asked engineering to explain a spike, the honest answer was "we're not sure," and that answer is what turns a cost conversation into a feature freeze.

There was also a dashboard, and everyone pointed at it as proof they were "doing FinOps." A dashboard shows you the number. It does not change anyone's behavior, and it definitely does not tell an engineer that the node group they oversized last sprint is the reason the graph bent upward. Visibility without attribution is just a prettier version of not knowing.

The last piece was fear. Every proposed saving came with "will this break production?" and without data nobody could answer, so nothing happened. The goal was to make cost a normal, reversible engineering decision instead of a quarterly emergency, and to do it without touching the delivery pipeline the team depended on.

Constraints

A few limits shaped the whole approach.

No feature freeze. Delivery velocity was the business. Any optimization that slowed shipping was off the table.
No reliability regressions. Saving money by removing redundancy or headroom was not a real saving.
Keep the delivery model. Full GitOps, dev to staging to production promotion, and 25% canary rollouts all had to stay exactly as they were.
Data stays isolated. The data tier runs in subnets with no internet route, and that boundary was non-negotiable.
Respect confidentiality. Real dollar figures stay private, so success is reported as percentages and unit economics.

Architecture

The SaaS runs entirely on a single EKS cluster per environment, across three availability zones. Each VPC has three tiers of subnets: three public subnets for ingress and NAT, three private subnets for the EKS worker nodes, and three isolated subnets with no internet route for the data tier. Everything the product needs runs in the cluster.

Compute is split into purpose-built managed node groups rather than one big pool, which is what makes both scheduling and cost control tractable. There are separate node groups for the frontend, the backend, data and ETL work, observability, the GitOps controllers, and the internal dashboards, plus a Spot-backed group for batch and preview workloads. Karpenter handles just-in-time node provisioning on top, so capacity follows demand instead of sitting idle.

The whole platform toolchain lives in the cluster, scheduled onto those node groups: Keycloak for single sign-on across the dashboards, Argo CD, and Grafana; SonarQube and Trivy in the delivery path; Argo CD and Argo Rollouts for GitOps and progressive delivery; and Prometheus with Grafana for observability.

The data tier (RDS with a multi-AZ standby, plus ElastiCache) lives in the isolated subnets. Those subnets have no NAT and no internet gateway route, so the databases cannot reach the internet and the internet cannot reach them. Nodes talk to them only over private VPC routes.

Cost work fails when it lives only in a finance spreadsheet, so the first move was to make spend attributable. A small, enforced tag taxonomy flows into AWS Cost Categories, which maps raw line items to teams and products, and from there into Budgets, Cost Anomaly Detection, and per-team showback.

The taxonomy was deliberately small. Five mandatory tags, not twenty, because a taxonomy nobody follows is worse than none. Environment, CostCenter, Application, and Owner covered almost every question we needed to answer, and ManagedBy flagged anything created by hand instead of through code. On EKS those tags also propagate to node groups and volumes, so cluster compute is attributable per team, not lumped under one anonymous bill.

Enforcement matters more than intent, so the tags were governed centrally. AWS Organizations Tag Policies defined the allowed keys and values, Service Control Policies blocked non-compliant resources, and consolidated billing plus the Cost and Usage Report gave one clean view across every account.

Chargeback is tempting, but it needs near-perfect tagging and it starts turf wars early. We started with showback: show each team its own spend, let central finance keep paying the bill, and move to chargeback only once the tags were trustworthy. Awareness drove most of the savings before any money changed hands internally.

Delivery: GitOps and progressive rollout

None of the cost work was allowed to disturb delivery, so it helps to see what delivery looks like. Every change runs through the same GitOps pipeline. CI builds and tests, SonarQube enforces a quality gate, and Trivy scans the image and the IaC. Only a clean build pushes to ECR and bumps the image digest in the GitOps manifests repository. Argo CD notices the change and syncs it to the cluster.

Nothing goes fully live at once. Argo Rollouts takes over at the cluster and shifts traffic in steps, starting at 25%, then pausing to check analysis metrics before it widens. If the metrics stay healthy it promotes; if they degrade it rolls back on its own, with no human in the loop. That single behavior is what let the cost changes ship safely, because a right-sized deployment that misbehaved would be caught at 25% of traffic, not 100%.

Environments follow the same path every time. A feature or preview environment spins up per pull request on the Spot-backed node group, merges auto-deploy to dev, the same image digest promotes to staging for integration tests, and only then does it reach production behind the canary.

The canary itself is a few lines of Argo Rollouts config, and it is the same for a feature change or a cost change.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: frontend
spec:
  strategy:
    canary:
      steps:
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 100

Implementation

The baseline was code. Rather than tag resources by hand, every provider inherited a default set of tags, so new infrastructure was attributable from the moment it existed.

provider "aws" {
  region = "us-east-1"

  default_tags {
    tags = {
      Environment = "Prod"
      CostCenter  = "1001"
      ManagedBy   = "Terraform"
    }
  }
}

Then came the guardrails, automated so nobody had to remember to check a dashboard. A monthly budget with a forecast alert catches planned overspend before the month ends, and Cost Anomaly Detection catches the surprise 3am spike.

aws budgets create-budget \
  --account-id 111122223333 \
  --budget '{
    "BudgetName": "MonthlyCost",
    "BudgetLimit": { "Amount": "50000", "Unit": "USD" },
    "TimeUnit": "MONTHLY",
    "BudgetType": "COST"
  }'

aws ce create-anomaly-monitor \
  --anomaly-monitor '{
    "MonitorName": "CoreServices",
    "MonitorType": "DIMENSIONAL",
    "MonitorDimension": "SERVICE"
  }'

With attribution and guardrails in place, the actual optimization ran as normal GitOps changes, each behind the canary, each with a one-commit rollback.

Find the waste. Use Cost Explorer grouped by the new tags, plus Kubernetes right-sizing signals from the metrics stack, to rank the most over-provisioned node groups and workloads.
Right-size node groups in reversible steps. Drop one instance size or one replica at a time, ship it through the 25% canary, and watch the SLOs. The old manifest is one revert away.
Let Karpenter consolidate. Enable consolidation so underused nodes are drained and replaced with fewer, better-packed ones, and move interruptible and preview work to the Spot node group.
Put dev and staging to sleep. Scale non-production node groups to zero overnight and on weekends. Nothing runs when nobody is working.
Commit last, not first. Only after cluster usage was stable did we buy Compute Savings Plans, so we committed to real baseline usage rather than to waste.

The commitment step is where teams most often lose money, by chasing the deepest discount for a workload they are about to change. The rule we used was simple: match the commitment to the roadmap, not to the current instance.

The team was midway through moving several backend services to Graviton for better price-performance. A three-year EC2 Instance Savings Plan on the old family would have looked cheaper on paper and then stranded the moment those services migrated. A Compute Savings Plan gave up a few points of discount but stayed flexible across families, regions, Fargate, and Lambda, which matters even more on EKS where node groups change shape often. That small premium was cheap insurance.

Results

Over two quarters, the monthly bill came down by roughly 41%, and delivery never paused. Every cost change went out through the same 25% canary as any feature. The breakdown, as representative shares of the total reduction, looked like this.

Lever	Share of the saving	Nature of the change
Right-sizing node groups + Karpenter consolidation	Largest	Reversible, canary-guarded
Scheduling dev and staging to sleep	Large	Fully reversible
Compute Savings Plans matched to the roadmap	Meaningful	1-year, flexible
Spot for batch, preview, and CI	Meaningful	Interruption-tolerant only
Storage cleanup and lifecycle policies	Smaller	One-time plus ongoing

The more durable result was cultural. Cost stopped being a quarterly fire drill. Teams could see their own node-group spend, cost showed up in pull requests as a normal signal, and the "will this break?" fear faded because every change had a canary and a rollback. Treat the percentage as illustrative and the mechanics as the real deliverable.

Lessons

Attribution is the whole game. Nothing else worked until spend had an owner, because you cannot optimize a shared cluster you cannot see per team. The five-tag taxonomy, enforced in code and propagated to node groups, paid for itself before a single node was resized.

Progressive delivery is what makes cost work safe. On a normal deploy model, right-sizing production feels risky enough that teams avoid it. With a 25% canary and automated rollback, a bad resize is a non-event, so the team actually did the work instead of flinching.

Commit to usage, not to hope. The most expensive mistake in cloud cost work is a long, rigid commitment bought early to chase a headline discount. Buy commitments after usage is stable, prefer flexibility while the architecture is still moving, and treat the discount rate as secondary to not stranding the plan.

If I did it again, I would wire a pull-request cost estimate in on day one. Putting the number in front of the engineer at the moment they change a manifest moved behavior more than any dashboard did.

Frequently Asked Questions

A freeze trades a measurable problem for an unmeasurable one. You save some money and lose delivery velocity, customer momentum, and team morale, none of which show up cleanly on the bill. Almost all of the saving here came from waste and mismatched commitments, not from doing less, so the freeze would have hurt the business while barely touching the real cost drivers.

Treat it like any other change. Drop one size or one replica at a time, ship it through the same 25% Argo Rollouts canary as a feature, and watch the SLOs during the pause windows. Let Karpenter consolidate underused nodes rather than doing it by hand. Because every step is a GitOps commit, the rollback is a one-line revert, so a bad resize is caught at 25% of traffic and reverted, not discovered in a postmortem.

Twenty-five percent is a deliberate balance. It is a big enough slice that real traffic patterns and enough metric volume show up quickly, so the analysis step can make an honest call, but small enough that a bad release only touches a quarter of users before it rolls back. Smaller first steps are reasonable for very high-risk changes, but 25% gave this team fast, trustworthy signal without much blast radius.

The isolated subnets have no NAT and no internet gateway route, so the databases cannot reach the internet and vice versa. The application nodes in the private subnets reach RDS and ElastiCache over private VPC routes only. Anything the data tier genuinely needs from an AWS service goes through VPC endpoints, which keep that traffic on the AWS network rather than the public internet.

It makes it safer, which in practice makes it happen. Every cost change is a normal pull request that flows dev to staging to production behind the canary, with SonarQube and Trivy gates on the way. There is no separate risky "cost project," just ordinary changes with the same guardrails as everything else, so teams approve them quickly.

Enforced tagging. Until spend is attributable per team, product, environment, and node group, every other optimization is guesswork. A small mandatory taxonomy, applied through Terraform default tags and AWS Organizations tag policies, turns the bill from one opaque number into a map you can act on.

References

QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog

Mohammad Abu Mattar — Wed, 15 Jul 2026 09:00:00 GMT

When Bitnami moved its long-trusted catalog behind a paid tier, thousands of teams woke up to a supply-chain problem they didn't choose. The free images they had pinned in production would stop getting updates, and the migration clock started that morning. QuenchWorks is my answer to that: a from-scratch, security-first catalog of container images and Helm charts, built entirely from source, hardened under a strict zero-CVE build gate, signed, and free. This is how it's put together and why each decision earns its place.

Impact

QuenchWorks is real and in production use, not a proof of concept. It replaced Bitnami for common workloads with a catalog that is built from source, provable, and free.

Everything below is verifiable: pull any image and check its signature, SBOM, and provenance yourself. The build gate stays green because the base is small enough that there is almost nothing to be vulnerable in, so the numbers hold instead of drifting the week after launch.

The problem

The Bitnami catalog was popular for good reasons. It was broad, it was versioned, and it was maintained well enough that most teams never thought about it. Its weaknesses only became obvious once access changed: you didn't control the build, you couldn't prove what was inside a given image, and continued free access was never actually guaranteed.

That last point is the one that bites. When an upstream catalog changes its terms, every image: line you pinned becomes a liability at once. You either pay, fork, or scramble. And even before that day comes, an opaque image is its own quiet risk. If you can't see how a layer was produced, you can't reason about what a scanner finds inside it, and you can't answer a security review with anything better than "we trust the vendor."

Most hardened-image alternatives fix one slice of this and charge for the rest. I wanted the whole thing: a catalog broad enough to actually replace Bitnami for common workloads, provable rather than "trust us," and free with no pull limits and no lock-in. If it couldn't be all three, it wasn't worth building.

Constraints

A handful of hard limits shaped every later decision.

Zero fixable CVEs, enforced by the build. Not a nightly report someone reads later. A gate that fails the build so a vulnerable image never ships in the first place.
Built from source. No repackaging of someone else's opaque binary layers. If it's in the image, we produced it.
Provable. Every image needs a bill of materials and build provenance that a consumer can verify without trusting me.
Free to run and maintain. The whole system builds on free CI, so cost can never be the reason it slips behind a paywall later.
Multi-arch. amd64 and arm64, because production is both now, not one or the other.

Those constraints pull against each other. Zero fixable CVEs across 150+ images sounds impossible if you picture a fat base image. Building everything from source sounds slow. The architecture is what makes them coexist.

Architecture

The catalog is a pipeline, not a pile of Dockerfiles. Each image is declared as an apko plus melange spec, built from source on Wolfi, scanned against a zero-fixable-CVE gate, signed, and only then published pinned by digest. Charts sit on a shared library chart and reference those images by digest.

The single decision that makes the zero-CVE gate realistic is the base. QuenchWorks builds on Wolfi, a glibc Linux undistro designed for containers. Most images start with no shell, no package manager, and a tiny set of packages. There's simply very little in the image that can be vulnerable, so keeping the gate green is a fight you can actually win instead of an endless race against a bloated base.

The image itself is assembled declaratively. melange builds signed APK packages from source, and apko composes those packages plus the Wolfi base into an OCI image with no Dockerfile involved. Because the whole thing is declared, the contents are known, reproducible, and easy to record as a bill of materials.

The zero-CVE gate and the minimal base are the same decision viewed twice. You don't reach zero fixable CVEs by patching harder. You reach it by shipping so little that there's almost nothing to patch.

A catalog is never done, though, because CVEs are disclosed against packages long after an image ships. So the pipeline runs in reverse on a schedule. A nightly Trivy rescan checks every published image, and when a fix lands upstream the affected image rebuilds, re-enters the gate, gets re-signed, and republishes under a new digest. The catalog trends toward zero drift without anyone babysitting it.

Implementation

Each image is a pair of specs. melange describes how to build the package from source, and apko describes how to assemble the final image. Here's the shape of both, trimmed for clarity.

package:
  name: my-app
  version: 1.2.3
environment:
  contents:
    packages:
      - build-base
pipeline:
  - uses: fetch
    with:
      uri: https://example.com/my-app-${{package.version}}.tar.gz
      expected-sha256: '...'
  - uses: autoconf/configure
  - uses: autoconf/make
  - uses: autoconf/make-install

contents:
  repositories:
    - https://packages.wolfi.dev/os
  packages:
    - my-app
    - ca-certificates-bundle
accounts:
  users:
    - username: nonroot
      uid: 65532
  run-as: 65532
archs:
  - x86_64
  - aarch64
entrypoint:
  command: /usr/bin/my-app

The gate is one Trivy call, and it's deliberately strict about what counts. It only fails on CVEs that have a fix available, because a vulnerability with no upstream patch isn't something a rebuild can clear. Everything fixable has to be at zero before the image is allowed out.

# Fail the build if any FIXABLE HIGH/CRITICAL vulnerability is present
trivy image --ignore-unfixed --severity HIGH,CRITICAL \
  --exit-code 1 ghcr.io/quenchworks/my-app:latest

Once an image passes, it gets signed and attested before it's pushed for real. Signing is keyless with Cosign, so there's no long-lived private key to leak, and the SBOM and SLSA provenance ride along as attestations.

Build from source. melange produces signed APKs; apko assembles the image with the Wolfi base, nonroot user, and read-only root filesystem defaults.
Gate on zero fixable CVEs. Trivy runs in comprehensive mode. One fixable HIGH or CRITICAL fails the pipeline, so a vulnerable image never reaches the registry.
Sign and attest. Cosign signs the image keyless, then attaches an SPDX SBOM and a SLSA build-provenance attestation.
Publish pinned by digest. The image is pushed, and the Helm charts reference it by sha256: digest, never by a movable tag.
Rescan nightly. A scheduled Trivy run watches for new fixes and triggers the self-heal rebuild loop.

On the delivery side, charts are the second half of the story. Every chart builds on a shared quench-common library chart, so common concerns like security context, probes, and labels live in one place instead of being copy-pasted 120 times. Each chart pins its image by digest.

Pinning by digest instead of tag is what makes the catalog trustworthy in practice. A tag can be moved; a digest can't. When a consumer pins a QuenchWorks chart, they get exactly the bytes that passed the gate.

image:
  repository: ghcr.io/quenchworks/postgresql
  # Pinned by digest, not tag. This is the exact image that passed the gate.
  digest: 'sha256:abc123...'

The proof only matters if consumers can check it, so verification is a first-class step, not an afterthought. Anyone can verify an image's signature and attestations before it runs, and an admission policy can enforce that in the cluster so unsigned or unverifiable images never schedule.

cosign verify \
  --certificate-identity-regexp '^https://github.com/quenchworks/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/quenchworks/postgresql@sha256:abc123...

Results

The catalog is real and in use, not a proof of concept.

150+ container images built from source on Wolfi, each gated to zero fixable CVEs.
120+ production Helm charts on the shared quench-common library, every one pinned to its image by digest.
Every image cosign-signed with an SPDX SBOM and SLSA provenance, published under an ArtifactHub verified-publisher organization.
Multi-arch (amd64 and arm64), nonroot, and read-only root filesystem by default.
A nightly rescan and self-heal rebuild loop that keeps the catalog current as upstream ships fixes.
Free and independent. No subscription, no registry pull limits, no lock-in.

The counts above are current catalog figures. Per-image build times and scan times vary by package, so I'd treat any single number there as indicative rather than a benchmark.

Lessons

The biggest lesson is that the base image choice decides everything downstream. Trying to reach zero CVEs on a fat base is a treadmill; starting from Wolfi's minimal surface turns the gate into something you can keep green for months. If I'd started anywhere else, the self-heal loop would be firing constantly and the whole thing would feel like bailing water.

The second lesson is that provenance costs far less than it's worth. Signing and generating SBOMs added very little build time, but they change the catalog's whole posture. It stops being "trust me" and becomes "verify it yourself," which is the entire point of replacing an opaque upstream. If I were doing it again, I'd wire verification into the consumer docs even earlier, because an unverified signed image is only half the value.

The one thing I'd watch more carefully next time is chart sprawl. The quench-common library chart paid for itself immediately, but library conventions need to be locked down early. Once a few charts drift from the shared patterns, every future change gets more expensive.

Frequently Asked Questions

It's zero fixable CVEs, and it's mostly a consequence of the base. Wolfi images ship with almost nothing beyond what the app needs, so there's very little surface for a vulnerability to live in. The Trivy gate then fails any build with a fixable HIGH or CRITICAL, so a vulnerable image can't ship. Vulnerabilities with no upstream fix are tracked but don't block, because a rebuild can't clear them.

A tag is a movable pointer; a digest is the content itself. If you pin :latest or even :1.2.3, the bytes behind that tag can change. Pinning sha256:... guarantees you get exactly the image that passed the gate and was signed. It's the difference between "probably the right image" and "provably the right image."

Use cosign verify with the QuenchWorks certificate identity and OIDC issuer, as shown above. That checks the keyless signature against the transparency log. You can also verify the SBOM and SLSA provenance attestations, and enforce all of it in-cluster with an admission policy so nothing unsigned ever schedules.

The nightly Trivy rescan catches it. If the CVE is fixable, the affected image rebuilds from source, goes back through the gate, gets re-signed, and republishes under a new digest. You pick up the fix by moving your pin to the new digest. Nobody has to notice the CVE manually for the loop to run.

It's free, with no subscription and no registry pull limits. The catch, if you call it one, is that you verify and pin things yourself rather than outsourcing trust to a vendor relationship. That's a feature for most teams: you get provenance you can audit instead of a support contract you have to believe.

Yes. The charts and images are independent. You can pull a single hardened image by digest, or install one chart, without buying into everything. The quench-common library chart is an implementation detail of the charts, not something you have to adopt in your own repos.