Summary
The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer laptop. The last one was a genuine incident.
An image in a production-bound pod had been built from a fork by a misconfigured pipeline in a sandbox account. Nothing about it was malicious. The admission check denied it because it could not prove the image was built by the pipeline, from the commit it claimed, which is exactly the question the gate exists to ask.
Under the previous setup, that image runs, and nobody finds out.
Impact
No user-facing impact, which is the point of catching it at admission rather than after. The real exposure was what the old system could not see:
- builds ran on long-lived shared EC2 runners, where a job could read files another job left behind
- registry credentials were static access keys stored as CI variables, rotated when someone remembered, and a compromised build in a low-value repo held the same credentials as everything else
- deployments referenced mutable tags, so anyone with push access could
change what
prodmeant on any cluster, with no review
Timeline
There is no clock, and there would not have been one under the old setup either. The detection method was the gate itself: the pod was denied at admission, the denial was logged, and the 20/2/1 breakdown of all 23 denials comes from those admission logs.
Root cause
The image came from a misconfigured pipeline in a sandbox account, building from a fork of the repository rather than the repository. Nothing in the old deployment path would have surfaced that: the manifest named a tag, the registry had the tag, and the cluster pulled it.
Contributing factors
Two builds of the same tag were not the same image. While investigating something unrelated, we found a 40MB difference between two builds of the same tag, a week apart. Dependencies resolved at build time from public registries by floating version range. An image with no stable identity cannot have a provenance story worth checking.
Nobody could answer the auditor’s question. The program started from a fair question: can you prove the container running in production is the one your pipeline built from the commit you think it was? We could tell a plausible story. We could not produce evidence.
The blast radius of one credential was everything. Static, shared, rarely rotated: the fork-built image was benign, but the same trust model would have carried a hostile one just as quietly.
What changed
The full program is the case study: hermetic builds, signed provenance, digest pinning, and the admission gate. The pieces this incident proved out:
- Admission enforcement with telemetry first. Two weeks of logging verdicts and denying nothing found 11 problems that would each have been a production incident on enforcement day.
- Reproducible builds. Rebuilding the same commit later produced the same digest for 52 of 60 services; the eight exceptions embed build timestamps and are on the backlog.
- No static registry credentials. Eleven before, zero after, measured by secret scan.
- A documented two-approver bypass. Expected to be abused, used twice in six weeks, both times legitimately during an incident.
Did it hold
Six weeks of post-enforcement telemetry kept the measurement honest: images deployable without provenance went from all to zero, services deploying by mutable tag from 60 of 60 to 0 of 60, and “which commit is running” went from a twenty-minute archaeology exercise to one command. The break-glass procedure was rehearsed twice in game days and used zero times in anger.
What I would do differently
Trust an image you cannot reproduce. The 40MB divergence was found by accident while investigating something else; reproducibility should have been a standing check, not a discovery.
The full program, including the nine-week rollout and the measured results, is written up in Hardening a CI/CD Supply Chain to SLSA Level 3.
