--- title: "Postmortem: A Provenance Gate Caught a Fork-Built Container Image" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/incidents/post/slsa-admission-gate-fork-built-image --- [Home](/)›[Incidents](/incidents)›[All Categories](/incidents/categories)›[Security](/incidents/categories/security) Incidents [Security](/incidents/categories/security)[CI/CD](/incidents/categories/cicd) # The Admission Gate Denied an Image Built from a Fork [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 20 Sept 202603 Mins read05 Mins listen [Markdown for AI(opens in a new tab)](/incidents/post/slsa-admission-gate-fork-built-image/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR - An admission gate verifying build provenance denied 23 pods in its first six weeks: twenty stale base images, two hand-built images, and one genuine incident, an image built from a fork by a misconfigured pipeline in a sandbox account. - Nothing malicious. Under the old setup it would have run in production and nobody would have known. - The same audit found a 40MB difference between two builds of the same tag. > The gate did not catch an attack. It caught the absence of proof. ### The Admission Gate Denied an Image Built from a Fork Contents [Summary](#summary)[Impact](#impact)[Timeline](#timeline)[Root cause](#root-cause)[Contributing factors](#contributing-factors)[What changed](#what-changed)[Did it hold](#did-it-hold)[What I would do differently](#what-i-would-do-differently) ## [Summary](#summary) The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer laptop. The last one was a genuine incident. An image in a production-bound pod had been built from a fork by a misconfigured pipeline in a sandbox account. Nothing about it was malicious. The admission check denied it because it could not prove the image was built by the pipeline, from the commit it claimed, which is exactly the question the gate exists to ask. Under the previous setup, that image runs, and nobody finds out. ## [Impact](#impact) No user-facing impact, which is the point of catching it at admission rather than after. The real exposure was what the old system could not see: - builds ran on long-lived shared EC2 runners, where a job could read files another job left behind - registry credentials were static access keys stored as CI variables, rotated when someone remembered, and a compromised build in a low-value repo held the same credentials as everything else - deployments referenced mutable tags, so anyone with push access could change what `prod` meant on any cluster, with no review ## [Timeline](#timeline) There is no clock, and there would not have been one under the old setup either. The detection method was the gate itself: the pod was denied at admission, the denial was logged, and the 20/2/1 breakdown of all 23 denials comes from those admission logs. ## [Root cause](#root-cause) The image came from a misconfigured pipeline in a sandbox account, building from a fork of the repository rather than the repository. Nothing in the old deployment path would have surfaced that: the manifest named a tag, the registry had the tag, and the cluster pulled it. ## [Contributing factors](#contributing-factors) **Two builds of the same tag were not the same image.** While investigating something unrelated, we found a 40MB difference between two builds of the same tag, a week apart. Dependencies resolved at build time from public registries by floating version range. An image with no stable identity cannot have a provenance story worth checking. **Nobody could answer the auditor’s question.** The program started from a fair question: can you prove the container running in production is the one your pipeline built from the commit you think it was? We could tell a plausible story. We could not produce evidence. **The blast radius of one credential was everything.** Static, shared, rarely rotated: the fork-built image was benign, but the same trust model would have carried a hostile one just as quietly. ## [What changed](#what-changed) The full program is the case study: hermetic builds, signed provenance, digest pinning, and the admission gate. The pieces this incident proved out: 1. **Admission enforcement with telemetry first.** Two weeks of logging verdicts and denying nothing found 11 problems that would each have been a production incident on enforcement day. 2. **Reproducible builds.** Rebuilding the same commit later produced the same digest for 52 of 60 services; the eight exceptions embed build timestamps and are on the backlog. 3. **No static registry credentials.** Eleven before, zero after, measured by secret scan. 4. **A documented two-approver bypass.** Expected to be abused, used twice in six weeks, both times legitimately during an incident. ## [Did it hold](#did-it-hold) Six weeks of post-enforcement telemetry kept the measurement honest: images deployable without provenance went from all to zero, services deploying by mutable tag from 60 of 60 to 0 of 60, and “which commit is running” went from a twenty-minute archaeology exercise to one command. The break-glass procedure was rehearsed twice in game days and used zero times in anger. ## [What I would do differently](#what-i-would-do-differently) Trust an image you cannot reproduce. The 40MB divergence was found by accident while investigating something else; reproducibility should have been a standing check, not a discovery. The full program, including the nine-week rollout and the measured results, is written up in [Hardening a CI/CD Supply Chain to SLSA Level 3](/case-studies/post/cicd-supply-chain-slsa-level-3). Was this useful? ## Tags [#Postmortem](/incidents/tags/postmortem)[#SLSA](/incidents/tags/slsa)[#Supply Chain](/incidents/tags/supply-chain)[#Kubernetes](/incidents/tags/kubernetes)[#Admission Control](/incidents/tags/admission-control)[#Provenance](/incidents/tags/provenance) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image&title=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork&summary=An%20image%20built%20from%20a%20fork%20by%20a%20misconfigured%20pipeline%20in%20a%20sandbox%20account%20reached%20a%20production%20cluster%20and%20was%20denied%20at%20admission.%20Under%20the%20old%20setup%20it%20would%20have%20run%2C%20and%20nobody%20would%20have%20known.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork%20https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image&text=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image&title=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image&t=The%20Admission%20Gate%20Denied%20an%20Image%20Built%20from%20a%20Fork "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fslsa-admission-gate-fork-built-image&media=&description=An%20image%20built%20from%20a%20fork%20by%20a%20misconfigured%20pipeline%20in%20a%20sandbox%20account%20reached%20a%20production%20cluster%20and%20was%20denied%20at%20admission.%20Under%20the%20old%20setup%20it%20would%20have%20run%2C%20and%20nobody%20would%20have%20known. "Share on Pinterest")[Email]() ## Comments ## Continue on this topic The same subject, covered a different way from the article above. - [ ![](/_astro/hero.CMArPlpu_Z2r9qz9.webp) Case study ### Hardening a CI/CD Supply Chain to SLSA Level 3 How we made a build pipeline tamper-evident end to end: hermetic builds, KMS-signed provenance, and an admission gate that refuses anything it cannot verify. ](/case-studies/post/cicd-supply-chain-slsa-level-3) - [ ![](/_astro/hero.xI_32nOg_Zby34O.webp) Article ### What's the Deal with Shift-Left Security, and Why Should You Care? How to implement shift-left security with SAST, DAST, and SCA in your CI/CD pipeline to cut costs and catch issues earlier, with tool examples like SonarQube, Trivy, and OWASP ZAP. ](/blog/post/shift-left-security-sast-dast-sca-cicd) - [ Cheatsheet ### GitHub Actions The workflow YAML you write over and over. Triggers, jobs and steps, secrets, matrix builds, caching, artifacts, and reusable workflows in one reference. ](/cheatsheets/github-actions) - [ ![](/_astro/hero.D4Pcicvc_Gb0yT.webp) Dev tip ### ArgoCD GitOps: Sync Kubernetes Deployments Automatically from Git Stop running kubectl apply by hand. This dev tip shows how ArgoCD watches a Git repo and keeps your Kubernetes cluster matching it, with automatic sync, self-heal, and drift detection. ](/devtips/post/argocd-gitops-kubernetes-deployments-git-sync) ## You might also enjoy More posts on similar topics ## [A Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Architecture Summary A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region #Postmortem#AWS#Multi Region+3 tags [read more](/incidents/post/single-region-payments-api-regional-outage) 1 related post