From Blue-Green to Progressive Delivery with Argo Rollouts

The incident that started this was not caused by carelessness. It was caused by someone being careful.

A release went out at 14:20 on a Tuesday. Error rate climbed within ninety seconds. The on-call engineer saw the alert, opened the dashboard, checked whether it was a blip, decided it was not, found the previous image tag, and rolled back. Total time from deploy to recovery: five minutes and ten seconds.

Every step in that sequence was correct. Nobody panicked, nobody guessed, nobody skipped a check. And for five minutes, every single user got 500s, because our deploy process had exactly one setting: all of them, now.

Impact

Twenty-seven services moved from all-at-once deployments to canary releases with automated analysis against live metrics. The rollback stopped being a runbook step performed by a person and became a state transition performed by a controller, which took it from minutes to under a minute without anyone deciding anything.

The number that mattered internally was not the rollback time. It was the share of users who saw a bad release at all. Under the old process that was everyone, every time. Under the new one a failed release reaches roughly one user in twenty for about a minute. The figures below were measured on our own systems over the fourteen-week rollout and the quarter that followed, so read them as the shape to expect rather than a promise.

That last figure needs a caveat we insisted on internally. Change failure rate fell partly because failures were caught before reaching everyone, and partly because the analysis step made people write metrics they had not previously bothered with. Some of the improvement is better deployment and some of it is better observability, and we could not cleanly separate the two.

The problem

We had a deploy process, and it was fine, in the way that a process is fine right up until the moment you measure what it costs.

Every service was a standard Kubernetes Deployment with a RollingUpdate strategy. ArgoCD synced from Git, the deployment rolled pods over about ninety seconds, and if something was wrong you found out from an alert and reverted the commit. Reverting was reliable and slow, because reliability came from a human confirming it was warranted.

Three properties of that setup produced most of our incidents.

A RollingUpdate is not a safety mechanism. It replaces pods gradually, which people read as gradual exposure. It is not. Once the first new pod is Ready, the Service load balances across old and new indiscriminately, so a share of traffic hits the new version immediately and that share grows regardless of whether the new version is healthy. Readiness probes catch a pod that failed to start. They say nothing about a pod that starts perfectly and returns the wrong answer.

The rollback decision was the slow part, not the rollback. kubectl rollout undo is fast. Deciding to run it is not. The engineer has to distinguish a real regression from ordinary variance, which means opening a dashboard and looking at a graph, which takes as long as it takes.

Nobody wanted to deploy on a Friday. That is a soft cost that turns into a hard one. Work batched up over the weekend, Monday releases were larger, and larger releases fail more often, which reinforced the fear that produced them.

Constraints

We were not starting from a clean sheet, and three constraints shaped every decision.

A fourth constraint emerged during the pilot and turned out to matter more than the first three. Several services had no useful per-version metrics at all. They reported aggregate error rate for the service, with no label distinguishing which pods served which request, which makes canary analysis impossible in principle rather than in practice.

Architecture

Argo Rollouts replaces the Deployment resource with a Rollout resource that has the same pod template and a strategy that can pause, measure and decide. The controller manages two ReplicaSets and shifts traffic between them by manipulating the service selectors, or by writing to an ingress or service mesh when one is present.

We considered blue-green first, because it is simpler and Argo Rollouts supports both. We rejected it for a specific reason.

Blue-green runs the new version alongside the old at full capacity, then switches 100% of traffic at once. It makes rollback instant, which is genuinely valuable. It does nothing about blast radius, because when the switch happens every user is on the new version. A bad release still reaches everyone; it just reaches them for less time.

The other half of the architecture is the analysis. An AnalysisTemplate defines metric queries and success conditions, and the Rollout references it at each step. The controller runs the queries, evaluates the conditions, and either proceeds or aborts.

Implementation

The Rollout resource

The migration from a Deployment was mostly mechanical. Same spec.template, different kind, plus a strategy block.

1
apiVersion: argoproj.io/v1alpha1
2
kind: Rollout
3
metadata:
4
  name: checkout
5
spec:
6
  replicas: 12
7
  revisionHistoryLimit: 3
8
  selector:
9
    matchLabels:
10
      app: checkout
11
  template:
12
    metadata:
13
      labels:
14
        app: checkout
15
    spec:
16
      containers:
17
        - name: checkout
18
          image: registry.internal/checkout:v2.14.0
19
          ports:
20
            - containerPort: 8080
21
          readinessProbe:
22
            httpGet: {path: /healthz/ready, port: 8080}
23
            periodSeconds: 5
24
  strategy:
25
    canary:
26
      canaryService: checkout-canary
27
      stableService: checkout-stable
28
      trafficRouting:
29
        nginx:
30
          stableIngress: checkout
31
      analysis:
32
        templates:
33
          - templateName: canary-health
34
        startingStep: 1
35
        args:
36
          - name: service-name
37
            value: checkout
38
      steps:
39
        - setWeight: 5
40
        - pause: {duration: 5m}
41
        - setWeight: 25
42
        - pause: {duration: 5m}
43
        - setWeight: 50
44
        - pause: {duration: 3m}
45
        - setWeight: 100

Three details in there took us longer than they should have.

startingStep: 1 means the analysis begins after the first weight change rather than immediately, which avoids an analysis run against a canary that has no traffic yet. revisionHistoryLimit: 3 matters more than usual here, because an abort needs the previous ReplicaSet to still exist. And the two Services are required: Argo Rollouts manages their selectors itself, so they must exist and must not be managed by anyone else.

The analysis, which is where the actual thinking is

This is the part that determines whether the whole exercise is worth anything.

1
apiVersion: argoproj.io/v1alpha1
2
kind: AnalysisTemplate
3
metadata:
4
  name: canary-health
5
spec:
6
  args:
7
    - name: service-name
8
  metrics:
9
    - name: error-rate-vs-stable
10
      interval: 60s
11
      count: 5
12
      # Two consecutive failures abort. One is usually noise.
13
      failureLimit: 1
14
      consecutiveErrorLimit: 2
15
      provider:
16
        prometheus:
17
          address: http://prometheus.monitoring:9090
18
          query: |
19
            sum(rate(http_requests_total{
20
              service="{{args.service-name}}", version="canary", code=~"5.."
21
            }[2m]))
22
            /
23
            sum(rate(http_requests_total{
24
              service="{{args.service-name}}", version="canary"
25
            }[2m]))
26
      # Compared against the stable version over the same window, not
27
      # against a fixed number. The baseline moves with reality.
28
      successCondition: >-
29
        result[0] <= 1.5 *
30
        (sum(rate(http_requests_total{service="{{args.service-name}}",
31
         version="stable", code=~"5.."}[2m])) /
32
         sum(rate(http_requests_total{service="{{args.service-name}}",
33
         version="stable"}[2m])) + 0.001)
34

35
    - name: p99-latency-vs-stable
36
      interval: 60s
37
      count: 5
38
      failureLimit: 1
39
      provider:
40
        prometheus:
41
          address: http://prometheus.monitoring:9090
42
          query: |
43
            histogram_quantile(0.99, sum by (le) (
44
              rate(http_request_duration_seconds_bucket{
45
                service="{{args.service-name}}", version="canary"
46
              }[2m])
47
            ))
48
      successCondition: result[0] <= 1.3 * {{args.stable-p99}}
49

50
    - name: minimum-traffic
51
      interval: 60s
52
      count: 5
53
      failureLimit: 0
54
      provider:
55
        prometheus:
56
          address: http://prometheus.monitoring:9090
57
          query: |
58
            sum(rate(http_requests_total{
59
              service="{{args.service-name}}", version="canary"
60
            }[2m])) * 120
61
      # Below this, the other two metrics are measuring noise.
62
      successCondition: result[0] >= 200

That third metric was added in week nine and is the one we would keep if we could only keep one.

Our first version measured error rate for thirty seconds at 5% weight. On a service handling forty requests a second, that is sixty requests in the canary. Sixty requests cannot distinguish a 0.5% error rate from a 2% one, so the analysis passed on statistical noise and we shipped two regressions through a canary that had approved them. The minimum-traffic metric turns “we did not observe a problem” into “we observed enough to have a view”, and it fails the analysis when the sample is too small rather than passing it.

The + 0.001 in the first success condition is similarly unglamorous and similarly load-bearing. Without it, a stable error rate of exactly zero makes the threshold zero, so a single 500 in the canary aborts the release. Adding a small floor means the comparison is “meaningfully worse than stable” rather than “not literally perfect”.

Wiring it into GitOps

The constraint that ArgoCD stays authoritative shaped this more than anything else. Argo Rollouts is designed for it, and there was one sharp edge.

1
apiVersion: argoproj.io/v1alpha1
2
kind: Application
3
metadata:
4
  name: checkout
5
spec:
6
  project: platform
7
  source:
8
    repoURL: [email protected]:platform/apps.git
9
    targetRevision: main
10
    path: apps/checkout
11
  destination:
12
    server: https://kubernetes.default.svc
13
    namespace: checkout
14
  syncPolicy:
15
    automated:
16
      prune: true
17
      selfHeal: true
18
  ignoreDifferences:
19
    # Argo Rollouts owns these selectors and rewrites them during a
20
    # rollout. Without this, selfHeal fights the rollout controller
21
    # and the canary flaps between weights.
22
    - group: ''
23
      kind: Service
24
      jsonPointers:
25
        - /spec/selector

1
# Fully automatic: every step gated by analysis, no human in the path.
2
steps:
3
  - setWeight: 5
4
  - pause: {duration: 5m}
5
  - setWeight: 50
6
  - pause: {duration: 5m}
7
  - setWeight: 100
8

9
# Manual gate: pause indefinitely until someone promotes.
10
# We used this for the first two weeks of each service's migration.
11
steps:
12
  - setWeight: 5
13
  - pause: {} # requires: kubectl argo rollouts promote checkout
14
  - setWeight: 50
15
  - pause: {duration: 5m}
16
  - setWeight: 100

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: argo-rollouts-notification-configmap
5
data:
6
  trigger.on-rollout-aborted: |
7
    - when: rollout.status.abort == true
8
      send: [rollout-aborted]
9
  template.rollout-aborted: |
10
    slack:
11
      attachments: |
12
        [{
13
          "title": "Canary aborted: {{.rollout.metadata.name}}",
14
          "color": "#dc2626",
15
          "fields": [
16
            {"title": "Revision", "value": "{{.rollout.status.currentPodHash}}"},
17
            {"title": "Reason", "value": "analysis run failed"}
18
          ]
19
        }]

That ignoreDifferences block is the sharp edge. ArgoCD with selfHeal: true sees the Service selector that Argo Rollouts just rewrote, decides it has drifted from Git, and rewrites it back. The two controllers then fight, and the canary oscillates between weights while the rollout never completes. It presents as a mysterious flapping deploy and the fix is four lines.

The abort path

An abort is a state transition, not a command.

Nothing scaled down and nothing restarted. The stable ReplicaSet was still running at full capacity the whole time, so aborting is a selector change and traffic returns to the version that was already serving it. That is why the median is forty-one seconds rather than the time it takes to schedule pods.

Results

Fourteen weeks, six teams, twenty-seven services. We migrated in waves of three or four, and we deliberately started with a mid-traffic service rather than the least important one, because the least important service has too little traffic for canary analysis to work and would have taught us nothing.

Change failure rate over the following quarter fell 68%, from 11.2% of deploys causing a user-visible regression to 3.6%. Deploy frequency rose about 40%, which we did not plan for and which is the clearest evidence that the fear of Friday deploys was a real constraint rather than a cultural quirk.

The individual deploy got slower, from ninety seconds to about twelve minutes. Nobody complained, because a deploy that nobody has to watch takes zero minutes of anyone’s attention regardless of how long it runs.

Seven services never made it onto canary. Their traffic is low enough that a 5% slice over five minutes is a few dozen requests, and no amount of configuration makes that statistically meaningful. Putting them on canary would have produced an analysis that always passes, which is worse than no analysis because it looks like one.

Lessons

Compare against the incumbent, never against a constant. This is the single most important thing we learned and it cost us two weeks. A fixed threshold is wrong in both directions: it fires overnight when low traffic makes normal variance look extreme, and it stays quiet at peak when a genuine regression is still under the number. The stable version, measured over the same window, is the only baseline that moves with reality.

A canary with too little traffic is worse than no canary. It produces a green tick that means nothing, and people trust it. Measure the sample size explicitly and fail the analysis when it is too small, rather than letting the other metrics quietly assess noise.

The migration was blocked on observability, not on the tool. Argo Rollouts took an afternoon to install. Getting every service to emit metrics labelled by version took most of the first three weeks, and it was the actual project. If your services cannot distinguish their own versions in Prometheus, you do not have a deployment problem to solve yet.

Progressive delivery only catches what is visible in minutes. Three failures got through in the following quarter and every one of them was outside what a canary can see. A memory leak that took four hours to matter. A migration that was fine at 5% write volume and locked the table at 100%. A bug affecting only accounts created before a 2021 schema change, which is under 2% of users and never landed in the canary slice at all. Narrowing blast radius is real and it is not a substitute for testing.

Frequently Asked Questions

References

1. Argo Rollouts documentation
2. Canary deployment strategy
3. Analysis and progressive delivery
4. Traffic management with Argo Rollouts
5. Blue-green deployment strategy
6. ArgoCD ignoreDifferences
7. Google SRE Workbook: canarying releases
8. DORA: change failure rate
9. Expand and contract migration pattern