Secrets Management Migration to Vault Without an Outage

This project did not start with a security review. It started with a resignation.

A senior engineer gave notice, and someone asked the reasonable question of what they could still log into after their last day. Disabling the SSO account was easy. Answering the rest took nine days, because the honest answer was that we did not know. Database passwords lived in environment variables. A few lived in AWS Systems Manager Parameter Store. One lived in a CI variable that predated the current CI system. At least two lived in a Kubernetes Secret that had been committed to a repository, force-pushed away, and then quietly re-added because a deploy broke without it.

We could not answer the question, so we built something that could.

Impact

Forty-one services moved from static, indefinitely-lived credentials to dynamic ones that Vault creates on demand and destroys on a timer. No service took downtime during the migration, and no phase required a coordinated release across teams. The two phases that carried real risk were both reversible without a deploy.

The change that mattered most is not a number. It is that the resignation question now takes about thirty seconds to answer, because every read of every secret is in an audit log with the workload identity attached. The figures below were measured on our own systems during the eleven-week rollout, so treat them as the shape to expect rather than a promise.

The problem

Secrets had not been managed badly on purpose. They had accumulated. Every individual decision was reasonable at the time, and the sum of them was a system where nobody could enumerate what existed.

Here is roughly where things actually lived when we started counting:

Four problems came out of that inventory, and only one of them was the one we expected.

Constraints

We wrote these down before choosing anything, and they eliminated most of the appealing shortcuts.

Architecture

The mental shift is that Vault does not store your database password. For the paths that matter, there is no stored password at all. Vault holds an admin connection to the database and manufactures a user when something asks for one.

Two pieces do the work here, and they are worth naming separately.

The auth method solves identity. A pod already has a cryptographic proof of who it is, in the form of a projected ServiceAccount token, and Vault’s Kubernetes auth method verifies that token by calling TokenReview against the cluster API. This is the part that breaks the chicken-and-egg problem. You do not need a secret to get a secret, because the cluster already vouches for the pod.

The secrets engine solves lifetime. A database secrets engine holds one privileged connection and a SQL template. When a workload reads its credentials path, Vault runs that template, hands back a username and password it just created, and starts a lease clock. When the clock runs out, Vault runs the revocation SQL.

The migration shape

Getting the architecture right was the easy half. The interesting design work was the order of operations.

Implementation

Wiring the auth method

The Kubernetes auth method needs a ServiceAccount of its own that is allowed to call TokenReview, and then one role per workload.

# Vault verifies pod tokens by asking the cluster. This is the only
# long-lived credential in the whole design, and it can only read
# the TokenReview API.
vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# One role per workload. The bound_* fields are the authorization:
# only this ServiceAccount, in this namespace, gets this policy.
vault write auth/kubernetes/role/orders-api \
  bound_service_account_names=orders-api \
  bound_service_account_namespaces=prod \
  policies=orders-api-db \
  ttl=1h

The policy stays deliberately narrow. If a service needs two databases, it gets two paths, not a wildcard.

1
# orders-api-db policy. Read only, one path, no list.
2
# A wildcard here would quietly undo the whole point of the exercise.
3
path "database/creds/orders-api-ro" {
4
  capabilities = ["read"]
5
}

The database secrets engine

This is where credentials stop being things you store.

vault secrets enable database

vault write database/config/orders \
  plugin_name=postgresql-database-plugin \
  allowed_roles="orders-api-ro" \
  connection_url="postgresql://{{username}}:{{password}}@orders.internal:5432/orders?sslmode=verify-full" \
  username="vault_admin" \
  password="$VAULT_DB_ADMIN_PASSWORD"

# Immediately rotate the admin password so that even we do not know it.
# After this call the value in your shell history is dead.
vault write -f database/rotate-root/orders

That last command surprises people, so it is worth being explicit: after rotate-root, the database admin password exists only inside Vault. No human has it. If Vault is destroyed you recover by resetting it out of band, which is a tradeoff you should decide on deliberately rather than discover.

The role is the SQL template that gets run per request:

1
-- Vault substitutes a generated name and password, and stamps the
2
-- expiry from default_ttl. The DROP at revocation time is generated
3
-- from the same name, so cleanup needs no bookkeeping on our side.
4
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
5
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";

vault write database/roles/orders-api-ro \
  db_name=orders \
  [email protected] \
  default_ttl="8h" \
  max_ttl="24h"

Getting it into the pod without touching the app

This is the part that made the no-rewrite constraint achievable. The Vault Agent injector adds a sidecar via annotations, and the sidecar writes a file. Applications read files already.

1
# No application change, no SDK, no import. The injector adds an init
2
# container and a sidecar, and the app finds a file where it expects one.
3
metadata:
4
  annotations:
5
    vault.hashicorp.com/agent-inject: 'true'
6
    vault.hashicorp.com/role: 'orders-api'
7
    vault.hashicorp.com/agent-inject-secret-db: 'database/creds/orders-api-ro'
8
    vault.hashicorp.com/agent-inject-template-db: |
9
      {{- with secret "database/creds/orders-api-ro" -}}
10
      DB_USER={{ .Data.username }}
11
      DB_PASSWORD={{ .Data.password }}
12
      {{- end }}

1
// Phase 2 in fifteen lines. Prefer the rendered file, fall back to the
2
// environment, and never fail closed on a parse error. This shipped to
3
// every service before a single service actually depended on Vault.
4
func dbCreds() (user, pass string) {
5
    user, pass = os.Getenv("DB_USER"), os.Getenv("DB_PASSWORD")
6

7
    b, err := os.ReadFile("/vault/secrets/db")
8
    if err != nil {
9
        return user, pass // Vault absent or not ready. Carry on.
10
    }
11
    for _, line := range strings.Split(string(b), "\n") {
12
        k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
13
        if !ok || v == "" {
14
            continue // Half-written file. Keep what we had.
15
        }
16
        switch k {
17
        case "DB_USER":
18
            user = v
19
        case "DB_PASSWORD":
20
            pass = v
21
        }
22
    }
23
    return user, pass
24
}

Checking the shadow phase

Phase 1 is only worth two weeks if you actually look at the data. We ran this across every migrated pod daily and treated any output at all as a blocker.

That single mismatch is the reason the dual-read helper falls back instead of failing. The agent was not slow or broken. The app simply started faster than the sidecar, which is normal and which we would have shipped straight into production had we skipped the shadow phase.

Results

The migration ran eleven weeks and cost no downtime. Rotation stopped being an event. Dynamic credentials roll every eight hours by design, so the thing we used to avoid now happens roughly a thousand times a day without anyone scheduling it. The database also stopped seeing one shared user. Each pod connects as its own role, which means a slow query, a lock, or a bad migration now points at a specific workload instead of at all forty.

The audit answer is the result that changed how the team works. Every credential read is a log line carrying the workload identity, the path, and the time, so the question that started the project is now a query rather than an investigation. We tested this against a real departure in week nine and had a complete answer in under a minute, against nine days for the same question in January.

Lessons

The dual-read window is the entire trick. Everything expensive about a secrets migration comes from the moment where the old source is gone and the new one is not proven. A fallback that prefers the new path and tolerates its absence collapses that moment to nothing, and it costs about fifteen lines per language.

Do not let the rollback plan need a deploy. For three of four phases, undoing a mistake meant deleting a file or removing an annotation. That is what let us move a service a day instead of scheduling change windows. The final phase does need a deploy to reverse, which is precisely why it went last and went one service at a time.

Dynamic secrets promote your secrets manager to tier zero. This is the part that gets skipped in the pitch. Once a pod cannot start without Vault, Vault’s availability is your availability. We run three nodes across zones with Raft and auto-unseal, we test unseal quarterly, and we accepted deliberately that a total Vault outage stops new pods from starting. Decide that before phase 3, not during it.

Frequently Asked Questions

References

• Vault Kubernetes auth method
• Vault database secrets engine
• Vault Agent injector for Kubernetes
• Vault audit devices
• Kubernetes projected ServiceAccount tokens
• Kubernetes TokenReview API
• gitleaks, for auditing secrets in Git history