--- title: "Secrets Management Migration to Vault Without an Outage" description: "How 41 services moved from scattered environment variables and parameter stores to Vault with dynamic, short-lived credentials, using a four-phase dual-read rollout that never made Vault a hard startup dependency until the very last step." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/case-studies/post/secrets-migration-to-vault --- # Secrets Management Migration to Vault Without an Outage This project did not start with a security review. It started with a resignation. A senior engineer gave notice, and someone asked the reasonable question of what they could still log into after their last day. Disabling the SSO account was easy. Answering the rest took nine days, because the honest answer was that we did not know. Database passwords lived in environment variables. A few lived in AWS Systems Manager Parameter Store. One lived in a CI variable that predated the current CI system. At least two lived in a Kubernetes Secret that had been committed to a repository, force-pushed away, and then quietly re-added because a deploy broke without it. We could not answer the question, so we built something that could. ## Impact Forty-one services moved from static, indefinitely-lived credentials to dynamic ones that Vault creates on demand and destroys on a timer. No service took downtime during the migration, and no phase required a coordinated release across teams. The two phases that carried real risk were both reversible without a deploy. The change that mattered most is not a number. It is that the resignation question now takes about thirty seconds to answer, because every read of every secret is in an audit log with the workload identity attached. The figures below were measured on our own systems during the eleven-week rollout, so treat them as the shape to expect rather than a promise.
## The problem Secrets had not been managed badly on purpose. They had accumulated. Every individual decision was reasonable at the time, and the sum of them was a system where nobody could enumerate what existed. Here is roughly where things actually lived when we started counting: - infrastructure/ - k8s/ - base/ - secrets.yaml (committed, then removed, then re-added) - overlays/ - prod/ - patch-env.yaml (three passwords, inline) - terraform/ - modules/ - rds/ - main.tf (master password in a variable with a default) - envs/ - prod.tfvars (gitignored, and therefore also un-backed-up) - .github/ - workflows/ - deploy.yml (reads eleven repository secrets) - ssm/ - /prod/api/db-password (rotated once, in 2023) Four problems came out of that inventory, and only one of them was the one we expected. | Problem | Why it actually hurt | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | No inventory | We could not answer "what does this person have access to" without grepping every repo and asking three teams. | | No expiry | A credential leaked in 2023 was still valid in 2026. Nothing about the system noticed or cared. | | No attribution | The database saw one user called `api_prod` doing everything. Forty pods shared it, so a bad query had forty suspects. | | Rotation meant an outage | Changing a password meant editing a manifest and doing a rolling restart, in the right order, at the same time. We rotated roughly never. | The one we expected was leakage. The one that actually caused incidents was the last row. Because rotation was risky, we avoided it, and because we avoided it, every credential had years of exposure behind it. The lack of rotation was a consequence of the deployment design, not of anyone being careless. ## Constraints We wrote these down before choosing anything, and they eliminated most of the appealing shortcuts. 1. **No downtime, and no coordinated cutover.** Forty-one services across nine teams. Any plan requiring everyone to deploy on the same afternoon was not a plan. 2. **No application rewrites.** Services span Node.js, Go, and two Python codebases. Asking every team to add a Vault SDK would have taken a quarter and produced four different implementations. 3. **Rollback must not need a deploy**, at least until the final phase. If the fix for a bad migration step is "ship a hotfix to nine teams", the migration is not safe. 4. **Vault cannot become a hard startup dependency by accident.** It is allowed to become one deliberately, at the end, with eyes open. 5. **The audit log is a deliverable**, not a side effect. If we cannot answer the resignation question at the end, the project failed regardless of what else it achieved. ## Architecture The mental shift is that Vault does not store your database password. For the paths that matter, there is no stored password at all. Vault holds an admin connection to the database and manufactures a user when something asks for one. _How one database credential is issued_ Two pieces do the work here, and they are worth naming separately. The **auth method** solves identity. A pod already has a cryptographic proof of who it is, in the form of a projected ServiceAccount token, and Vault's Kubernetes auth method verifies that token by calling `TokenReview` against the cluster API. This is the part that breaks the chicken-and-egg problem. You do not need a secret to get a secret, because the cluster already vouches for the pod. The **secrets engine** solves lifetime. A database secrets engine holds one privileged connection and a SQL template. When a workload reads its credentials path, Vault runs that template, hands back a username and password it just created, and starts a lease clock. When the clock runs out, Vault runs the revocation SQL. The mapping is `namespace + ServiceAccount name` to a Vault role, and the role to a policy. That means your authorization model is your Kubernetes deployment topology, which you already maintain. We did not have to invent a second identity system, and that is most of why this fit in a quarter. ### The migration shape Getting the architecture right was the easy half. The interesting design work was the order of operations. _Four phases, and what breaks in each_ ## Implementation ### Wiring the auth method The Kubernetes auth method needs a ServiceAccount of its own that is allowed to call `TokenReview`, and then one role per workload. ```bash # Vault verifies pod tokens by asking the cluster. This is the only # long-lived credential in the whole design, and it can only read # the TokenReview API. vault auth enable kubernetes vault write auth/kubernetes/config \ kubernetes_host="https://kubernetes.default.svc:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt # One role per workload. The bound_* fields are the authorization: # only this ServiceAccount, in this namespace, gets this policy. vault write auth/kubernetes/role/orders-api \ bound_service_account_names=orders-api \ bound_service_account_namespaces=prod \ policies=orders-api-db \ ttl=1h ``` The policy stays deliberately narrow. If a service needs two databases, it gets two paths, not a wildcard. ```hcl # orders-api-db policy. Read only, one path, no list. # A wildcard here would quietly undo the whole point of the exercise. path "database/creds/orders-api-ro" { capabilities = ["read"] } ``` ### The database secrets engine This is where credentials stop being things you store. ```bash vault secrets enable database vault write database/config/orders \ plugin_name=postgresql-database-plugin \ allowed_roles="orders-api-ro" \ connection_url="postgresql://{{username}}:{{password}}@orders.internal:5432/orders?sslmode=verify-full" \ username="vault_admin" \ password="$VAULT_DB_ADMIN_PASSWORD" # Immediately rotate the admin password so that even we do not know it. # After this call the value in your shell history is dead. vault write -f database/rotate-root/orders ``` That last command surprises people, so it is worth being explicit: after `rotate-root`, the database admin password exists only inside Vault. No human has it. If Vault is destroyed you recover by resetting it out of band, which is a tradeoff you should decide on deliberately rather than discover. The role is the SQL template that gets run per request: ```sql -- Vault substitutes a generated name and password, and stamps the -- expiry from default_ttl. The DROP at revocation time is generated -- from the same name, so cleanup needs no bookkeeping on our side. CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}"; ``` ```bash vault write database/roles/orders-api-ro \ db_name=orders \ creation_statements=@orders-ro.sql \ default_ttl="8h" \ max_ttl="24h" ``` ### Getting it into the pod without touching the app This is the part that made the no-rewrite constraint achievable. The Vault Agent injector adds a sidecar via annotations, and the sidecar writes a file. Applications read files already. ```yaml # No application change, no SDK, no import. The injector adds an init # container and a sidecar, and the app finds a file where it expects one. metadata: annotations: vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/role: 'orders-api' vault.hashicorp.com/agent-inject-secret-db: 'database/creds/orders-api-ro' vault.hashicorp.com/agent-inject-template-db: | {{- with secret "database/creds/orders-api-ro" -}} DB_USER={{ .Data.username }} DB_PASSWORD={{ .Data.password }} {{- end }} ``` ```go // Phase 2 in fifteen lines. Prefer the rendered file, fall back to the // environment, and never fail closed on a parse error. This shipped to // every service before a single service actually depended on Vault. func dbCreds() (user, pass string) { user, pass = os.Getenv("DB_USER"), os.Getenv("DB_PASSWORD") b, err := os.ReadFile("/vault/secrets/db") if err != nil { return user, pass // Vault absent or not ready. Carry on. } for _, line := range strings.Split(string(b), "\n") { k, v, ok := strings.Cut(strings.TrimSpace(line), "=") if !ok || v == "" { continue // Half-written file. Keep what we had. } switch k { case "DB_USER": user = v case "DB_PASSWORD": pass = v } } return user, pass } ``` The `continue` on a malformed line is not defensive padding. The agent writes the file non-atomically on some versions, so a pod that starts during a render can genuinely read half a file. Failing closed there would have turned a harmless race into a crash loop, and we would have blamed Vault rather than the parser. ### Checking the shadow phase Phase 1 is only worth two weeks if you actually look at the data. We ran this across every migrated pod daily and treated any output at all as a blocker. _bash_ That single mismatch is the reason the dual-read helper falls back instead of failing. The agent was not slow or broken. The app simply started faster than the sidecar, which is normal and which we would have shipped straight into production had we skipped the shadow phase. ## Results The migration ran eleven weeks and cost no downtime. Rotation stopped being an event. Dynamic credentials roll every eight hours by design, so the thing we used to avoid now happens roughly a thousand times a day without anyone scheduling it. The database also stopped seeing one shared user. Each pod connects as its own role, which means a slow query, a lock, or a bad migration now points at a specific workload instead of at all forty. The audit answer is the result that changed how the team works. Every credential read is a log line carrying the workload identity, the path, and the time, so the question that started the project is now a query rather than an investigation. We tested this against a real departure in week nine and had a complete answer in under a minute, against nine days for the same question in January. | Before | After | How it was measured | | ---------------------------------------- | -------------------------------- | ------------------------------------------ | | Credential lifetime: indefinite | 8 hours, 24 hour ceiling | Vault lease TTL configuration | | Rotation frequency: roughly never | Continuous, no human involvement | Lease renewal counters | | Database users: 1 shared per service | 1 per pod | `pg_stat_activity` sampling | | Answering "what can they access": 9 days | Under 1 minute | Timed against a real departure, week 9 | | Static secrets in Git: 14 | 0 | `gitleaks` scan of full history at week 11 | ## Lessons **The dual-read window is the entire trick.** Everything expensive about a secrets migration comes from the moment where the old source is gone and the new one is not proven. A fallback that prefers the new path and tolerates its absence collapses that moment to nothing, and it costs about fifteen lines per language. **Do not let the rollback plan need a deploy.** For three of four phases, undoing a mistake meant deleting a file or removing an annotation. That is what let us move a service a day instead of scheduling change windows. The final phase does need a deploy to reverse, which is precisely why it went last and went one service at a time. The first time a pod could not start because Vault was busy, my honest reaction was that we had made things worse. Then I remembered that the alternative was a password from 2023 that nobody could rotate. I would rather debug an availability problem than not know what an ex-employee can log into. **Dynamic secrets promote your secrets manager to tier zero.** This is the part that gets skipped in the pitch. Once a pod cannot start without Vault, Vault's availability is your availability. We run three nodes across zones with Raft and auto-unseal, we test unseal quarterly, and we accepted deliberately that a total Vault outage stops new pods from starting. Decide that before phase 3, not during it. One thing we would do differently: we set the initial TTL to one hour because short felt safer. It generated enough renewal traffic to make the audit log genuinely hard to read, and it made every restart storm a small load test on Vault. Eight hours with a 24 hour ceiling gave the same practical security with a fraction of the noise. Short is not automatically better, and the right number depends on how fast you can revoke, not on how small you can make it. ## Frequently Asked Questions > **Why not just use a cloud provider secrets manager?** For static values, honestly, you should consider it. AWS Secrets Manager and SSM Parameter Store handle storage, encryption, and IAM-based access perfectly well, and they are less to run. We needed dynamic database credentials with per-pod identity across two clouds and one on-premises database, and that is where Vault's secrets engines pull ahead. If everything you have is in one cloud and your secrets are static, the managed option is the smaller answer and the smaller answer is usually right. > **What happens to running pods if Vault goes down?** Nothing immediately. A pod that already holds a credential keeps using it until the lease expires, so with an 8 hour TTL you have hours of headroom. What breaks is anything that needs a new credential: new pods, restarts, and scale-up events. That is a real availability coupling and it is the honest cost of phase 3. Plan for it with a highly available Vault, and know that a long outage combined with an autoscaling event is the scenario that actually hurts. > **How do you handle secrets for things that are not in Kubernetes?** Different auth method, same model. We use AppRole for a handful of virtual machines and AWS IAM auth for Lambda functions. The pattern stays identical: prove an identity the platform already issues, receive a token bound to a policy, read a path. Only step 3 in the issuance flow changes. Resist inventing a bespoke auth path per system, because the thing you are trying to remove is exactly that kind of sprawl. > **Do you really rotate the database admin password so nobody knows it?** Yes, and it is worth thinking about before you run it. After `rotate-root`, the only holder of that credential is Vault. That is exactly what you want for day-to-day security and exactly what you need a plan for during disaster recovery. Ours is documented: a break-glass procedure that resets the password through the cloud provider's control plane, requiring two approvers. Write that runbook before you rotate, not after. > **Was the audit log actually useful, or just a compliance checkbox?** Genuinely useful, though not in the way we expected. We assumed it was for investigations. In practice its most common use is answering "is anything still reading this old path", which is how we knew a decommission was safe. Ship the audit device to your log pipeline on day one, because the value comes from having history when you finally need it, and you cannot backfill it. > **How long would this take for a smaller estate?** The eleven weeks were mostly waiting, not working. Two weeks of that was the shadow phase deliberately doing nothing, and three more were the dual-read soak. For ten services with one language, the same shape compresses to about three weeks without cutting any of the safety, because the soak periods exist to catch races and they do not get longer with more services. Do not compress the soaks. Compress the number of services in flight at once. ## References - [Vault Kubernetes auth method](https://developer.hashicorp.com/vault/docs/auth/kubernetes) - [Vault database secrets engine](https://developer.hashicorp.com/vault/docs/secrets/databases) - [Vault Agent injector for Kubernetes](https://developer.hashicorp.com/vault/docs/platform/k8s/injector) - [Vault audit devices](https://developer.hashicorp.com/vault/docs/audit) - [Kubernetes projected ServiceAccount tokens](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) - [Kubernetes TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) - [gitleaks, for auditing secrets in Git history](https://github.com/gitleaks/gitleaks)