Hardening a CI/CD Supply Chain to SLSA Level 3

The question that started this was from an auditor, and it was a fair one: can you prove that the container running in production is the one your pipeline built from the commit you think it was? We had branch protection, mandatory review, image scanning, the whole checklist. None of it answered the question. We could tell a plausible story about how an image probably got there. We could not produce evidence.

This is the work that closed that gap. The target was SLSA Level 3, which in practice means three things: builds run on hardened, ephemeral infrastructure the build itself cannot tamper with; every artifact carries signed provenance describing how it was made; and something downstream actually checks that provenance before the artifact runs.

The problem

The pipeline looked fine on a diagram and was full of holes in practice.

Builds ran on a pool of long-lived EC2 runners shared between repositories. A job could read files another job left behind, and a compromised build in a low-value repo had the same registry credentials as everything else. Those credentials were static access keys stored as CI variables, rotated when someone remembered.

Dependencies resolved at build time from public registries by floating version range. Two builds of the same commit, a week apart, produced different images. That is not a hypothetical: we found a 40MB difference between two builds of the same tag while investigating something unrelated.

Deployments referenced mutable tags. A Kubernetes manifest said image: app:prod, and whatever prod pointed to at pull time was what ran. Anyone with push access could change what prod meant, on any cluster, with no review and no audit trail beyond registry logs nobody read.

The honest summary: our security model was “we trust everyone who has access, and access is broad.” The auditor’s question exposed that we had no artifact-level integrity at all. Scanning tells you an image has known CVEs. It tells you nothing about whether the image is yours.

Constraints

A few things were fixed before we started, and they shaped every decision.

We were on AWS and staying there, so managed services were preferred over self-hosted equivalents. That ruled out running our own Fulcio and Rekor instances for this phase, and pushed us toward KMS-backed keys.

The team was four platform engineers, part time, against a nine-week window before the audit follow-up. That budget meant we could not hand-roll verification logic per service. Whatever we built had to be central and default-on.

The last constraint was cultural and mattered most: developers would not accept a change that made deploys slower or more mysterious. If the gate rejected something, the error message had to say exactly what failed and what to do about it. A gate that produces “admission webhook denied the request” and nothing else gets disabled within a week.

Architecture

The design splits into four stages with a hard boundary between “producing an artifact” and “being allowed to run it.” The build environment produces evidence. The runtime environment demands evidence. Neither trusts the other.

The build runs in CodeBuild inside a private subnet with no NAT gateway and no internet gateway. Everything it needs arrives through VPC endpoints: CodeArtifact for dependencies, ECR for base images, S3 for caches, KMS for signing. A build that tries to curl an arbitrary host fails, which is exactly the behaviour we want. This is the “hermetic” part of Level 3, and it is the control that stops a malicious dependency from phoning home during a build.

Signing uses an asymmetric KMS key. The private key never exists outside KMS, so a fully compromised build host can ask KMS to sign something but cannot steal the key and sign things later, or elsewhere. The key policy allows kms:Sign only from the build role, and only when the request comes through the VPC endpoint.

Provenance goes in two places. The attestation is attached to the image in ECR as an OCI referrer, which is what the admission webhook reads. A copy also lands in an S3 bucket with Object Lock in compliance mode and a one-year retention, which is what we hand an auditor. The two are the same document; the duplication is deliberate, because ECR lifecycle policies will eventually expire images and we did not want evidence expiring with them.

Verification happens at admission. The webhook resolves the image digest, fetches the attestation, verifies the KMS signature, then checks the claims inside it against policy: the builder ID must be our CodeBuild project ARN, the source repository must be on an allowlist, and the commit must exist on a protected branch. A failure denies the pod and writes a Security Hub finding.

Implementation

We rolled this out in five steps, and deliberately did the boring ones first. Steps one to three change nothing observable, which is what made them safe to do quickly.

The build step itself is unremarkable, which is the point. This is the signing portion of the buildspec:

1
# buildspec.yml (post_build stage)
2
post_build:
3
  commands:
4
    # Push first, then sign what the registry actually stored. Signing a
5
    # locally-computed digest and hoping the registry agrees is how you get
6
    # signatures that verify against nothing.
7
    - docker push "$REPO:$COMMIT_SHA"
8
    - |
9
      DIGEST=$(aws ecr describe-images \
10
        --repository-name "$ECR_REPO" \
11
        --image-ids imageTag="$COMMIT_SHA" \
12
        --query 'imageDetails[0].imageDigest' --output text)
13
    - echo "signing $REPO@$DIGEST"
14
    - cosign sign --key "awskms:///$KMS_KEY_ID" --yes "$REPO@$DIGEST"
15
    - |
16
      cosign attest --key "awskms:///$KMS_KEY_ID" --yes \
17
        --type slsaprovenance \
18
        --predicate provenance.json \
19
        "$REPO@$DIGEST"
20
    - |
21
      aws s3 cp provenance.json \
22
        "s3://$EVIDENCE_BUCKET/${ECR_REPO}/${DIGEST#sha256:}.json"

The provenance document is generated from the build environment rather than hand-written, because anything hand-written drifts:

#!/usr/bin/env bash
# generate-provenance.sh - runs inside CodeBuild, before signing.
set -euo pipefail

cat > provenance.json <<JSON
{
  "buildDefinition": {
    "buildType": "https://mkabumattar.com/slsa/codebuild/v1",
    "externalParameters": {
      "repository": "${CODEBUILD_SOURCE_REPO_URL}",
      "ref": "${GIT_REF}"
    },
    "internalParameters": {
      "buildProject": "${CODEBUILD_BUILD_ARN%:*}",
      "image": "${CODEBUILD_BUILD_IMAGE}"
    },
    "resolvedDependencies": $(jq -c '[.dependencies[]
      | {uri: .name, digest: {sha512: .integrity}}]' lockfile-normalised.json)
  },
  "runDetails": {
    "builder": { "id": "${CODEBUILD_BUILD_ARN%:*}" },
    "metadata": {
      "invocationId": "${CODEBUILD_BUILD_ID}",
      "startedOn": "${BUILD_START_TIME}"
    }
  }
}
JSON

# Fail the build rather than emit provenance that claims nothing.
jq -e '.buildDefinition.resolvedDependencies | length > 0' provenance.json >/dev/null

The verification policy is the piece worth reviewing carefully, because a policy that only checks “is there a signature” is theatre. A signature proves someone with the key signed it. The claims are what tell you who built what from where:

1
# policy.yaml - evaluated by the admission webhook
2
apiVersion: policy.sigstore.dev/v1beta1
3
kind: ClusterImagePolicy
4
metadata:
5
  name: require-slsa-provenance
6
spec:
7
  images:
8
    - glob: '123456789012.dkr.ecr.eu-west-1.amazonaws.com/**'
9
  authorities:
10
    - name: kms-release-key
11
      key:
12
        kms: awskms:///arn:aws:kms:eu-west-1:123456789012:key/PLACEHOLDER
13
      attestations:
14
        - name: must-have-slsa-provenance
15
          predicateType: slsaprovenance
16
          policy:
17
            type: cue
18
            data: |
19
              // Builder must be our CodeBuild project, not any signer.
20
              predicate: builder: id: "arn:aws:codebuild:eu-west-1:123456789012:project/release-builder"
21
              // Source must be a repo we own.
22
              predicate: invocation: configSource: uri: =~"^git\\+https://git-codecommit\\.eu-west-1\\.amazonaws\\.com/v1/repos/(platform|payments|web)-"

The IAM boundary deserves a note. Each pipeline stage assumes a distinct role through OIDC, and the signing role is the only one with kms:Sign. The build role cannot sign, and the signing stage cannot modify source. Splitting these means compromising the build does not get you a signing oracle for arbitrary content:

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": "kms:Sign",
7
      "Resource": "arn:aws:kms:eu-west-1:123456789012:key/PLACEHOLDER",
8
      "Condition": {
9
        "StringEquals": {
10
          "aws:SourceVpce": "vpce-PLACEHOLDER",
11
          "kms:MessageType": "DIGEST"
12
        }
13
      }
14
    }
15
  ]
16
}

Results

These numbers come from our own pipeline telemetry over the nine weeks plus the six weeks after enforcement, and I have marked what is measured against what is estimated.

The build slowdown is real and worth naming: about 38 seconds median, from signing, attestation, and pulling dependencies through an endpoint instead of a warm local cache. Nobody complained, but I would not have gotten away with 3 minutes.

Eight services still do not reproduce bit-for-bit. All eight embed a build timestamp somewhere in their output. That is a fixable problem and it is on the backlog, not a blocker, because provenance still tells you which build produced which digest even when a rebuild would differ.

The admission gate denied 23 pods in the first six weeks. Twenty were stale base images during the tail of the rollout. Two were hand-built images, which is exactly the case this was designed to catch. One was a genuine incident: an image built from a fork by a misconfigured pipeline in a sandbox account. Nothing malicious, but under the old setup it would have run in production and nobody would have known.

Lessons

Signing without verification is just metadata. We spent the first three weeks on signing infrastructure and it bought us nothing until the gate went in. If you are short on time, design the gate first and work backwards; it forces you to be honest about which claims you can actually check.

Dry-run mode is not optional. Two weeks of logging verdicts and denying nothing found 11 problems that would each have been a production incident on enforcement day. The temptation to skip it because “we already know what is deployed” was strong and would have been wrong.

Hermetic builds break things you did not know were network-dependent. A test suite that quietly hit a public API. A build script that fetched a shell script from a gist. A base image that ran apt-get update at build time. Each was a small fix, and each was also a real supply chain hole we would never have found by reading code.

The escape hatch keeps the control alive. We built a documented, audited, two-approver bypass and expected it to be abused. It was used twice in six weeks, both times legitimately, during an incident. Teams tolerate a strict gate when there is a visible, honest way through it. Without one, someone eventually disables the webhook at 3am and nobody turns it back on.

Do not chase the level number. SLSA levels are a useful shared vocabulary and a terrible goal. The valuable artifacts here were the row-by-row threat list and the denial messages, neither of which appears in the spec. We hit Level 3 for the container build path, and the Terraform that provisions the cluster is still Level 1. Saying that plainly is more useful than a badge.

Frequently Asked Questions

References

• SLSA specification, version 1.0
• SLSA build levels
• in-toto attestation framework
• Sigstore cosign documentation
• Signing with AWS KMS in cosign
• AWS Signer developer guide
• Amazon ECR support for OCI artifacts and referrers
• Sigstore policy-controller for Kubernetes admission
• Kubernetes admission webhook documentation
• S3 Object Lock
• CNCF Software Supply Chain Best Practices white paper