---
title: "Hardening a CI/CD Supply Chain to SLSA Level 3"
description: "How we made a build pipeline tamper-evident end to end: hermetic builds, KMS-signed provenance, and an admission gate that refuses anything it cannot verify."
author: "Mohammad Abu Mattar"
canonical: https://mkabumattar.com/case-studies/post/cicd-supply-chain-slsa-level-3
---

# Hardening a CI/CD Supply Chain to SLSA Level 3

The question that started this was from an auditor, and it was a fair one: can you prove that the container running in production is the one your pipeline built from the commit you think it was? We had branch protection, mandatory review, image scanning, the whole checklist. None of it answered the question. We could tell a plausible story about how an image probably got there. We could not produce evidence.

This is the work that closed that gap. The target was SLSA Level 3, which in practice means three things: builds run on hardened, ephemeral infrastructure the build itself cannot tamper with; every artifact carries signed provenance describing how it was made; and something downstream actually checks that provenance before the artifact runs.

## The problem

The pipeline looked fine on a diagram and was full of holes in practice.

Builds ran on a pool of long-lived EC2 runners shared between repositories. A job could read files another job left behind, and a compromised build in a low-value repo had the same registry credentials as everything else. Those credentials were static access keys stored as CI variables, rotated when someone remembered.

Dependencies resolved at build time from public registries by floating version range. Two builds of the same commit, a week apart, produced different images. That is not a hypothetical: we found a 40MB difference between two builds of the same tag while investigating something unrelated.

Deployments referenced mutable tags. A Kubernetes manifest said `image: app:prod`, and whatever `prod` pointed to at pull time was what ran. Anyone with push access could change what `prod` meant, on any cluster, with no review and no audit trail beyond registry logs nobody read.

The honest summary: our security model was "we trust everyone who has access, and access is broad." The auditor's question exposed that we had no artifact-level integrity at all. Scanning tells you an image has known CVEs. It tells you nothing about whether the image is yours.

_Attack paths before and after_

## Constraints

A few things were fixed before we started, and they shaped every decision.

No pipeline freeze. About 60 services deployed through this pipeline, 30 to 50 times a day between them. Any migration had to run alongside the existing flow, per service, with an easy way back.

We were on AWS and staying there, so managed services were preferred over self-hosted equivalents. That ruled out running our own Fulcio and Rekor instances for this phase, and pushed us toward KMS-backed keys.

The team was four platform engineers, part time, against a nine-week window before the audit follow-up. That budget meant we could not hand-roll verification logic per service. Whatever we built had to be central and default-on.

The last constraint was cultural and mattered most: developers would not accept a change that made deploys slower or more mysterious. If the gate rejected something, the error message had to say exactly what failed and what to do about it. A gate that produces "admission webhook denied the request" and nothing else gets disabled within a week.

## Architecture

The design splits into four stages with a hard boundary between "producing an artifact" and "being allowed to run it." The build environment produces evidence. The runtime environment demands evidence. Neither trusts the other.

_Hardened build and deploy pipeline on AWS_

The build runs in CodeBuild inside a private subnet with no NAT gateway and no internet gateway. Everything it needs arrives through VPC endpoints: CodeArtifact for dependencies, ECR for base images, S3 for caches, KMS for signing. A build that tries to `curl` an arbitrary host fails, which is exactly the behaviour we want. This is the "hermetic" part of Level 3, and it is the control that stops a malicious dependency from phoning home during a build.

Signing uses an asymmetric KMS key. The private key never exists outside KMS, so a fully compromised build host can ask KMS to sign something but cannot steal the key and sign things later, or elsewhere. The key policy allows `kms:Sign` only from the build role, and only when the request comes through the VPC endpoint.

Provenance goes in two places. The attestation is attached to the image in ECR as an OCI referrer, which is what the admission webhook reads. A copy also lands in an S3 bucket with Object Lock in compliance mode and a one-year retention, which is what we hand an auditor. The two are the same document; the duplication is deliberate, because ECR lifecycle policies will eventually expire images and we did not want evidence expiring with them.

Deploying by digest is the control that makes all of this worth anything. If manifests reference mutable tags, an attacker does not need to defeat your signature checks. They repoint the tag and wait for the next pod restart. We rewrote every manifest to `image: repo@sha256:...` before turning on verification.

Verification happens at admission. The webhook resolves the image digest, fetches the attestation, verifies the KMS signature, then checks the claims inside it against policy: the builder ID must be our CodeBuild project ARN, the source repository must be on an allowlist, and the commit must exist on a protected branch. A failure denies the pod and writes a Security Hub finding.

_Signing and admission handshake_

## Implementation

We rolled this out in five steps, and deliberately did the boring ones first. Steps one to three change nothing observable, which is what made them safe to do quickly.

1. **Move dependency resolution behind CodeArtifact.** Point every package manager at an upstream-backed CodeArtifact repository and commit lockfiles everywhere. This alone made builds reproducible enough that the later provenance meant something. It also surfaced 14 packages nobody could explain, three of which were unused.

2. **Rebuild the build environment as ephemeral and networkless.** New CodeBuild projects in a private subnet, VPC endpoints for the four services builds actually need, and a security group with no egress rules. Run both pipelines in parallel per service and diff the outputs until they match.

3. **Generate provenance without enforcing it.** Emit an in-toto attestation from every build, sign it with KMS, attach it to the image, and copy it to S3. Nothing verifies yet. This is the observation phase, and it is where we found that 9 of 60 services were building from a directory that was not the repository root, which would have failed verification later.

4. **Deploy the webhook in dry-run mode.** Verify every pod, deny nothing, log the verdict. Two weeks of this produced a list of every image that would have been blocked. Most were base images from before step three; two were a genuine surprise, built on a developer laptop and pushed by hand.

5. **Enforce, namespace by namespace.** Turn the webhook from audit to enforce, starting with the namespace the platform team owns, then non-production, then production in order of ascending traffic. Keep a documented per-namespace escape hatch requiring a second approver.

The build step itself is unremarkable, which is the point. This is the signing portion of the buildspec:

```yaml
# buildspec.yml (post_build stage)
post_build:
  commands:
    # Push first, then sign what the registry actually stored. Signing a
    # locally-computed digest and hoping the registry agrees is how you get
    # signatures that verify against nothing.
    - docker push "$REPO:$COMMIT_SHA"
    - |
      DIGEST=$(aws ecr describe-images \
        --repository-name "$ECR_REPO" \
        --image-ids imageTag="$COMMIT_SHA" \
        --query 'imageDetails[0].imageDigest' --output text)
    - echo "signing $REPO@$DIGEST"
    - cosign sign --key "awskms:///$KMS_KEY_ID" --yes "$REPO@$DIGEST"
    - |
      cosign attest --key "awskms:///$KMS_KEY_ID" --yes \
        --type slsaprovenance \
        --predicate provenance.json \
        "$REPO@$DIGEST"
    - |
      aws s3 cp provenance.json \
        "s3://$EVIDENCE_BUCKET/${ECR_REPO}/${DIGEST#sha256:}.json"
```

The provenance document is generated from the build environment rather than hand-written, because anything hand-written drifts:

```bash
#!/usr/bin/env bash
# generate-provenance.sh - runs inside CodeBuild, before signing.
set -euo pipefail

cat > provenance.json < 0' provenance.json >/dev/null
```

The verification policy is the piece worth reviewing carefully, because a policy that only checks "is there a signature" is theatre. A signature proves someone with the key signed it. The claims are what tell you _who built what from where_:

```yaml
# policy.yaml - evaluated by the admission webhook
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-slsa-provenance
spec:
  images:
    - glob: '123456789012.dkr.ecr.eu-west-1.amazonaws.com/**'
  authorities:
    - name: kms-release-key
      key:
        kms: awskms:///arn:aws:kms:eu-west-1:123456789012:key/PLACEHOLDER
      attestations:
        - name: must-have-slsa-provenance
          predicateType: slsaprovenance
          policy:
            type: cue
            data: |
              // Builder must be our CodeBuild project, not any signer.
              predicate: builder: id: "arn:aws:codebuild:eu-west-1:123456789012:project/release-builder"
              // Source must be a repo we own.
              predicate: invocation: configSource: uri: =~"^git\\+https://git-codecommit\\.eu-west-1\\.amazonaws\\.com/v1/repos/(platform|payments|web)-"
```

Write the denial message before you write the policy. Ours resolves the failed CUE constraint into a sentence like `image not admitted: provenance builder id is "…/laptop-build", expected "…/release-builder" (see go/slsa-denied)`. Support load on the rollout was close to zero, and that message is the only reason.

The IAM boundary deserves a note. Each pipeline stage assumes a distinct role through OIDC, and the signing role is the only one with `kms:Sign`. The build role cannot sign, and the signing stage cannot modify source. Splitting these means compromising the build does not get you a signing oracle for arbitrary content:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kms:Sign",
      "Resource": "arn:aws:kms:eu-west-1:123456789012:key/PLACEHOLDER",
      "Condition": {
        "StringEquals": {
          "aws:SourceVpce": "vpce-PLACEHOLDER",
          "kms:MessageType": "DIGEST"
        }
      }
    }
  ]
}
```

- pipeline/
  - buildspec.yml
  - generate-provenance.sh
  - policy/
    - **policy.yaml**
    - denial-messages.yaml
  - terraform/
    - kms.tf
    - iam.tf
    - codebuild.tf
    - vpc-endpoints.tf
- charts/
  - admission-webhook/
    - values.yaml
    - values-enforce.yaml

## Results

These numbers come from our own pipeline telemetry over the nine weeks plus the six weeks after enforcement, and I have marked what is measured against what is estimated.

| Measure                                         | Before   | After             | Source                      |
| ----------------------------------------------- | -------- | ----------------- | --------------------------- |
| Images deployable without provenance            | all      | 0                 | measured, admission logs    |
| Services deploying by mutable tag               | 60 of 60 | 0 of 60           | measured, manifest audit    |
| Static registry credentials in CI               | 11       | 0                 | measured, secret scan       |
| Median build duration                           | 4m 10s   | 4m 48s            | measured, CodeBuild metrics |
| Rebuild of same commit produces same digest     | no       | 52 of 60 services | measured                    |
| Time to answer "where did this image come from" | hours    | one query         | estimated                   |

The build slowdown is real and worth naming: about 38 seconds median, from signing, attestation, and pulling dependencies through an endpoint instead of a warm local cache. Nobody complained, but I would not have gotten away with 3 minutes.

Eight services still do not reproduce bit-for-bit. All eight embed a build timestamp somewhere in their output. That is a fixable problem and it is on the backlog, not a blocker, because provenance still tells you which build produced which digest even when a rebuild would differ.

The admission gate denied 23 pods in the first six weeks. Twenty were stale base images during the tail of the rollout. Two were hand-built images, which is exactly the case this was designed to catch. One was a genuine incident: an image built from a fork by a misconfigured pipeline in a sandbox account. Nothing malicious, but under the old setup it would have run in production and nobody would have known.

  The part I did not expect was how much easier incident response got. "Which
  commit is running" used to be a twenty minute archaeology exercise. Now it is
  one command.

## Lessons

**Signing without verification is just metadata.** We spent the first three weeks on signing infrastructure and it bought us nothing until the gate went in. If you are short on time, design the gate first and work backwards; it forces you to be honest about which claims you can actually check.

**Dry-run mode is not optional.** Two weeks of logging verdicts and denying nothing found 11 problems that would each have been a production incident on enforcement day. The temptation to skip it because "we already know what is deployed" was strong and would have been wrong.

**Hermetic builds break things you did not know were network-dependent.** A test suite that quietly hit a public API. A build script that fetched a shell script from a gist. A base image that ran `apt-get update` at build time. Each was a small fix, and each was also a real supply chain hole we would never have found by reading code.

**The escape hatch keeps the control alive.** We built a documented, audited, two-approver bypass and expected it to be abused. It was used twice in six weeks, both times legitimately, during an incident. Teams tolerate a strict gate when there is a visible, honest way through it. Without one, someone eventually disables the webhook at 3am and nobody turns it back on.

**Do not chase the level number.** SLSA levels are a useful shared vocabulary and a terrible goal. The valuable artifacts here were the row-by-row threat list and the denial messages, neither of which appears in the spec. We hit Level 3 for the container build path, and the Terraform that provisions the cluster is still Level 1. Saying that plainly is more useful than a badge.

## Frequently Asked Questions

> **Why KMS instead of keyless signing with Fulcio and Rekor?**

Keyless signing is a better long-term answer, because a short-lived certificate tied to a workload identity beats a long-lived key you have to protect. We chose KMS for this phase because it was already in our account, the key policy could be expressed in the IAM we already reviewed, and we did not have the appetite to run a transparency log before the audit. The cosign commands are nearly identical, so switching later is a change to one flag and the policy's authority block.

> **Does the admission webhook become a single point of failure?**

It can, and the default configuration is dangerous. A webhook with `failurePolicy: Fail` that becomes unavailable blocks every pod creation in the cluster, including the webhook's own replacement pods. We run three replicas across availability zones with a pod disruption budget, exclude the webhook's own namespace and `kube-system` from the policy, and keep the image cached on every node. We also rehearsed the recovery: a documented, break-glass procedure that flips to `Ignore`, which we have tested twice in game days and used zero times in anger.

> **How much does the signing and attestation add to build time?**

For us, about 38 seconds at the median, and that splits roughly into 6 seconds for the two KMS signing calls, 9 seconds for pushing and attaching the attestation, and the rest from resolving dependencies through a VPC endpoint rather than a local cache. The KMS portion is nearly constant regardless of image size, because you sign a digest rather than the image itself. If your builds get much slower than that, the cause is usually the network topology change, not the cryptography.

> **What happens to images built before you turned this on?**

They fail verification, because they have no attestation. We handled this by rebuilding every actively deployed service through the new pipeline during the dry-run phase, which took a coordinated afternoon. For images that are deployed rarely, the dry-run logs told us exactly which ones would break, so we could rebuild them ahead of enforcement rather than discovering it during an urgent deploy. There is no honest shortcut here: an unsigned legacy image either gets rebuilt or gets an audited exception.

> **Is SLSA Level 3 achievable without Kubernetes?**

Yes, and the build side is identical. What changes is where verification happens. On ECS you can verify in a deployment pipeline step before updating the task definition, or use a Lambda hooked to the deployment event. On EC2 with an AMI pipeline you verify before the AMI is marked as shared. The pattern is the same: find the last point where you control whether the artifact runs, and verify there. The Kubernetes admission webhook is convenient because it is a natural chokepoint, not because it is required.

> **Do you verify third-party images too?**

Not with the same policy, because we cannot demand our builder ID from an upstream image. We run a second policy for a small allowlist of external registries that requires a valid upstream signature where the publisher provides one, and pins by digest in all cases. Everything else is mirrored into our own ECR through a review step, and once mirrored it gets our attestation describing the mirroring, which is honest about what it proves: we vouch that this is the bytes we reviewed, not that we know how upstream built them.

> **How do you handle emergency deploys when the gate blocks something?**

The escape hatch is a label on the namespace that the policy excludes, applied through a short-lived pull request that requires a second approver and expires automatically after four hours via a controller that strips the label. Every use writes to CloudWatch and raises a Security Hub finding, so it is visible rather than quiet. Making it slightly awkward and completely visible has worked better than making it impossible, which just moves the bypass somewhere we cannot see.

## References

- [SLSA specification, version 1.0](https://slsa.dev/spec/v1.0/)
- [SLSA build levels](https://slsa.dev/spec/v1.0/levels)
- [in-toto attestation framework](https://github.com/in-toto/attestation)
- [Sigstore cosign documentation](https://docs.sigstore.dev/cosign/overview/)
- [Signing with AWS KMS in cosign](https://docs.sigstore.dev/cosign/key_management/signing_with_self_managed_keys/)
- [AWS Signer developer guide](https://docs.aws.amazon.com/signer/latest/developerguide/Welcome.html)
- [Amazon ECR support for OCI artifacts and referrers](https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html)
- [Sigstore policy-controller for Kubernetes admission](https://docs.sigstore.dev/policy-controller/overview/)
- [Kubernetes admission webhook documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
- [S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
- [CNCF Software Supply Chain Best Practices white paper](https://github.com/cncf/tag-security/blob/main/community/resources/security-whitepaper/v2/CNCF_SSCP_v1.pdf)