Speeding Up CI Pipelines: Caching, Parallelism, and Skipping Unnecessary Work

A slow pipeline does not just waste compute. It changes behaviour. Once feedback takes longer than about ten minutes, people stop waiting for it. They context-switch, batch up changes into bigger pull requests, and start merging on optimism. The cost shows up later as harder debugging, not as a line on your CI bill.

The tips below are in the order I would actually apply them, which is not the order they usually get applied. Measurement first, then eliminating work, then parallelising what remains, and only then caching. Most teams do this backwards and start with caching because it feels like the obvious knob.

Tip 1: measure before you touch anything

You cannot optimise what you have not timed. Every CI system exposes per-step duration somewhere, and almost nobody looks at it before making changes.

# GitHub Actions: average duration per step across the last 20 runs
gh run list --workflow ci.yml --limit 20 --json databaseId --jq '.[].databaseId' \
  | while read -r id; do
      gh api "repos/:owner/:repo/actions/runs/$id/jobs" \
        --jq '.jobs[] | .steps[] | select(.completed_at != null) |
              [.name,
               ((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601))
              ] | @tsv'
    done \
  | awk -F'\t' '{sum[$1]+=$2; n[$1]++}
                END {for (s in sum) printf "%6.1fs avg  %3d runs  %s\n", sum[s]/n[s], n[s], s}' \
  | sort -rn

The output usually surprises people. A pipeline everyone describes as “slow because of tests” turns out to spend nine minutes on a Docker build and four on tests.

Tip 2: do not build what cannot have broken

In a monorepo this is the single biggest win available, and it beats caching by a wide margin. If the change touched only services/billing, there is no reason to test services/search.

1
jobs:
2
  changes:
3
    runs-on: ubuntu-latest
4
    outputs:
5
      services: ${{ steps.filter.outputs.changes }}
6
    steps:
7
      - uses: actions/checkout@v4
8
      - uses: dorny/paths-filter@v3
9
        id: filter
10
        with:
11
          # A change to shared/ or the lockfile invalidates everything, so it
12
          # is listed under every service. Forgetting this is how you ship a
13
          # broken shared library with a green pipeline.
14
          filters: |
15
            billing:
16
              - 'services/billing/**'
17
              - 'shared/**'
18
              - 'pnpm-lock.yaml'
19
            search:
20
              - 'services/search/**'
21
              - 'shared/**'
22
              - 'pnpm-lock.yaml'
23
            web:
24
              - 'apps/web/**'
25
              - 'shared/**'
26
              - 'pnpm-lock.yaml'
27

28
  test:
29
    needs: changes
30
    if: needs.changes.outputs.services != '[]'
31
    strategy:
32
      fail-fast: false
33
      matrix:
34
        service: ${{ fromJSON(needs.changes.outputs.services) }}
35
    runs-on: ubuntu-latest
36
    steps:
37
      - uses: actions/checkout@v4
38
      - run: pnpm --filter "./services/${{ matrix.service }}" test

For monorepos with a real dependency graph, let the build tool work out what is affected instead of maintaining path filters by hand:

# Everything affected by this branch's changes, and nothing else
pnpm nx affected --target=test --base=origin/main --head=HEAD

# See the graph it derived before trusting it
pnpm nx show projects --affected --base=origin/main

# --filter with a git range resolves dependents automatically
pnpm turbo run test --filter='...[origin/main]'

# Query the targets whose inputs changed, then test exactly those
bazel query "rdeps(//..., set($(git diff --name-only origin/main)))" \
  --output=label | xargs bazel test

Tip 3: shard tests by measured time, not alphabetically

Splitting a suite into four equal counts of files gives four very unequal durations. One shard finishes in 40 seconds and another takes 6 minutes, and your wall clock is 6 minutes. Balance on recorded timings instead.

1
// scripts/shard.mjs - split spec files into N shards of similar duration.
2
// Reads timings from a previous run's JSON report; falls back to file size.
3
import {readFileSync} from 'node:fs';
4
import {globSync} from 'node:fs';
5

6
const shardCount = Number(process.argv[2] ?? 4);
7
const shardIndex = Number(process.argv[3] ?? 0);
8

9
let timings = {};
10
try {
11
  timings = JSON.parse(readFileSync('.ci/test-timings.json', 'utf8'));
12
} catch {
13
  // First run, or the artifact expired. Size is a rough proxy.
14
}
15

16
const files = globSync('src/**/*.test.js')
17
  .map((f) => ({f, cost: timings[f] ?? readFileSync(f).length / 1000}))
18
  .sort((a, b) => b.cost - a.cost); // longest first
19

20
// Greedy bin packing: always add the next longest to the lightest shard.
21
const shards = Array.from({length: shardCount}, () => ({cost: 0, files: []}));
22
for (const item of files) {
23
  const lightest = shards.reduce((a, b) => (a.cost <= b.cost ? a : b));
24
  lightest.files.push(item.f);
25
  lightest.cost += item.cost;
26
}
27

28
process.stdout.write(shards[shardIndex].files.join('\n'));

Greedy longest-first bin packing is not optimal, but it lands within a few percent of balanced and takes ten lines. The upgrade path is a real solver, and you will not need it.

Tip 4: cache keys that actually hit

A cache that misses every run costs you the save time and gives nothing back. The key must change when the dependencies change and not otherwise.

1
- uses: actions/cache@v4
2
  with:
3
    path: |
4
      ~/.local/share/pnpm/store
5
      node_modules/.cache
6
    # Exact key: same lockfile, same OS, same Node major.
7
    key: deps-${{ runner.os }}-node20-${{ hashFiles('pnpm-lock.yaml') }}
8
    # Fallback: a slightly stale store still saves most of the download.
9
    restore-keys: |
10
      deps-${{ runner.os }}-node20-
11
      deps-${{ runner.os }}-

Three rules that cover most cache mistakes:

• Hash the lockfile, never package.json. package.json says ^4.2.0 and does not change when the resolved version does.
• Always provide restore-keys. A partial hit that needs to fetch three new packages beats a cold miss that fetches nine hundred.
• Never cache the output of the thing you are testing. Caching dist/ across commits means you eventually test a stale build and cannot reproduce it locally.

For container builds, the cache lives in the registry rather than the CI cache:

# BuildKit inline cache: the pushed image carries its own layer metadata,
# so the next build on a cold runner can still reuse layers.
docker buildx build \
  --cache-from "type=registry,ref=$ECR_REPO:buildcache" \
  --cache-to   "type=registry,ref=$ECR_REPO:buildcache,mode=max" \
  --tag        "$ECR_REPO:$GIT_SHA" \
  --push .

mode=max stores intermediate layers too, not just the final ones. It makes the cache bigger and hits far more often, which is the trade you want.

Tip 5: build once, reuse everywhere

A pipeline that compiles in the lint job, again in the test job, and again in the package job is paying three times for one result. Build once and pass it forward.

1
build:
2
  runs-on: ubuntu-latest
3
  steps:
4
    - uses: actions/checkout@v4
5
    - run: pnpm install --frozen-lockfile && pnpm build
6
    - uses: actions/upload-artifact@v4
7
      with:
8
        name: dist-${{ github.sha }}
9
        path: dist/
10
        retention-days: 1 # CI artifacts are not backups
11

12
test:
13
  needs: build
14
  runs-on: ubuntu-latest
15
  steps:
16
    - uses: actions/checkout@v4
17
    - uses: actions/download-artifact@v4
18
      with:
19
        name: dist-${{ github.sha }}
20
        path: dist/
21
    - run: pnpm test:integration # runs against dist/, does not rebuild

This also removes a class of bug where the artifact you tested is not the artifact you shipped, because now there is only one artifact.

Putting it together

The shape matters more than the specific services. Decide what to build, fan out over only that, share a cache across the fan-out, and join at the end. The same topology maps cleanly onto GitHub Actions, GitLab CI, or Buildkite.

What order to actually do this in

References

• Caching dependencies to speed up workflows (GitHub Actions)
• Using a matrix for your jobs (GitHub Actions)
• dorny/paths-filter action
• Docker BuildKit registry cache backend
• Nx affected commands
• Turborepo filtering by git range
• AWS CodeBuild build caching
• AWS CodeBuild batch builds