--- title: "Building an Internal Developer Platform on Backstage and GitOps" description: "How golden paths in Backstage, self-service software templates, and Argo CD let product teams create, build, and ship services without filing tickets to the platform team." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/case-studies/post/internal-developer-platform-backstage-gitops --- # Building an Internal Developer Platform on Backstage and GitOps Product teams were spending more time waiting on the platform team than building features. Spinning up a new service meant opening a ticket, waiting for someone to provision a repo, wire up CI, write Kubernetes manifests, and hook up deployment, and each of those handoffs added days. We built an internal developer platform that turned that whole sequence into a self-service golden path: Backstage for the portal and software templates, Git as the single source of truth, and Argo CD reconciling desired state into the clusters. A developer picks a template, fills a short form, and gets a working repo plus a running service, with policy and RBAC acting as guardrails rather than manual gates. ## Impact The headline change was that creating and shipping a service stopped being a ticket and became a form. New services that used to take teams the better part of a sprint to stand up now scaffold in minutes, and the first deploy happens on merge without anyone from the platform team touching it. The platform team moved from doing one-off deploys to maintaining the templates that everyone else uses. Adoption is the metric that actually matters here, because a platform nobody uses is just more software to run. Within the first quarter most new services were created through the golden path rather than by hand, which is the signal that the paved road was genuinely easier than going around it. These numbers are specific to this rollout and were measured on our own usage, so treat them as a shape to expect rather than a guarantee.
## The problem Every new service started the same way: a ticket. The platform team owned the repo templates, the CI config, the base Kubernetes manifests, and the deploy pipeline, so nothing shipped without them in the loop. That made sense when there were a handful of services, but it stopped scaling. The queue grew, context-switching killed the platform team's own roadmap, and product teams learned to batch requests, which made each one bigger and slower. The deeper issue was that knowledge lived in people's heads and in copy-pasted YAML. Two teams standing up similar services would end up with subtly different setups, because each one copied whatever the last project happened to do. There was no paved road, just a lot of dirt tracks that mostly worked. When something went wrong in one of those setups, debugging it meant reverse-engineering choices nobody remembered making. We wanted product teams to move without asking permission for routine work, while the platform team kept ownership of what "correct" looks like. That is the tension an internal developer platform exists to resolve. ## Constraints The platform had to satisfy a few hard constraints, and every design decision came back to them. - **Self-service by default.** The common case, creating and deploying a service, had to happen with zero tickets and no human in the platform team's loop. - **Git as the source of truth.** Every change to what runs in a cluster had to be a commit, so we get review, history, and a trivial rollback for free. No `kubectl apply` from laptops. - **Guardrails, not gates.** Policy and RBAC had to be enforced automatically. A human manually approving routine deploys would just recreate the ticket queue we were killing. - **Paved road, not a walled garden.** Teams with genuinely unusual needs had to be able to step off the golden path without the platform blocking them, as long as they still passed policy. ## Architecture The platform is three moving parts wired together by Git. Backstage is the front door, where developers discover services and kick off golden paths. Git holds both application code and the deployment config that describes desired state. Argo CD watches Git and reconciles that desired state into the Kubernetes clusters. Backstage never talks to the clusters to make changes; it only ever writes to Git, which keeps the whole system auditable. _Platform control plane_ The Backstage catalog models the world as a small set of entities, and understanding those makes the rest of the platform click. A `Template` describes a golden path: its input parameters as a JSON schema, and the steps it runs to scaffold a service. Each template produces a `Component`, which is an actual service owned by a `Group` (a team). Argo CD then manages an `Application` resource that points at the component's config in Git and syncs it to a cluster. _Catalog and deployment model (class view)_ The reason Git sits in the middle of everything is that it turns two hard problems, auditability and rollback, into one solved problem: version control. Every deploy is a diff you can read, and undoing a bad change is `git revert`, which Argo CD then reconciles back automatically. ## Implementation The heart of the platform is the scaffolding flow. When a developer picks a template and submits the form, Backstage's scaffolder renders a skeleton from the template's inputs, creates a repository, opens a pull request, and registers the new component in the catalog. That is the moment the ticket used to be filed; now it is a button. _Self-service scaffolding to deploy (sequence)_ A software template is a `Template` entity plus a skeleton directory. The parameters block is a JSON schema, so Backstage renders it as a validated form for free. The steps block is what runs when the form is submitted. ```yaml title="template.yaml (Backstage software template)" apiVersion: scaffolder.backstage.io/v1beta3 kind: Template metadata: name: node-service title: Node.js service (golden path) description: A production-ready Node.js service with CI, Helm, and GitOps wired up. spec: owner: group:platform type: service parameters: - title: Service details required: [name, owner] properties: name: title: Name type: string pattern: '^[a-z][a-z0-9-]{2,30}$' owner: title: Owning team type: string ui:field: OwnerPicker steps: - id: fetch name: Fetch skeleton action: fetch:template input: url: ./skeleton values: name: ${{ parameters.name }} owner: ${{ parameters.owner }} - id: publish name: Create repository action: publish:github input: repoUrl: github.com?owner=acme&repo=${{ parameters.name }} defaultBranch: main - id: register name: Register in catalog action: catalog:register input: repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }} catalogInfoPath: /catalog-info.yaml ``` The skeleton ships the boring, correct defaults so no team has to reinvent them: a `catalog-info.yaml` so the service shows up in the catalog, a CI workflow that builds and signs the container image, a Helm chart, and the Argo CD `Application` that ties it to a cluster. That last file is what turns a repo into something GitOps actually deploys. ```yaml title="argocd-application.yaml (scaffolded into the config repo)" apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: node-service-dev namespace: argocd spec: project: default source: repoURL: https://github.com/acme/config path: apps/node-service/dev targetRevision: main destination: server: https://kubernetes.default.svc namespace: node-service syncPolicy: automated: prune: true selfHeal: true ``` Because the deploy config lives in Git and Argo CD self-heals, the platform naturally behaves like a state machine. A service moves from scaffolded, to built, to deployed in dev, through a policy gate, and on to production, and every transition is a commit. Modeling it that way made it obvious where the guardrails belong. _Service lifecycle (state machine)_ Guardrails are enforced at two layers. RBAC in Backstage and in the clusters decides who can do what, and admission policy with OPA or Kyverno decides what is allowed to run at all. A policy that every workload must set resource limits, for example, is a Kyverno rule that rejects the deploy at admission rather than a checklist item in a review. ```yaml title="require-resource-limits.yaml (Kyverno policy)" apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-limits spec: validationFailureAction: Enforce rules: - name: check-limits match: any: - resources: kinds: ['Pod'] validate: message: 'CPU and memory limits are required.' pattern: spec: containers: - resources: limits: memory: '?*' cpu: '?*' ``` The repo layout keeps application code and deployment config separate, which is a deliberate GitOps choice: app repos change on every feature, config repos change on every deploy, and keeping them apart makes the deploy history readable. - config/ - apps/ - node-service/ - dev/ - kustomization.yaml - deployment.yaml - prod/ - kustomization.yaml - deployment.yaml - argocd/ - node-service-dev.yaml - node-service-prod.yaml Onboarding an existing service that predates the platform is a short runbook rather than a rebuild. 1. Add a `catalog-info.yaml` to the repo so Backstage discovers and indexes the service. 2. Move its deployment manifests into the config repo under a per-environment path. 3. Add an Argo CD `Application` pointing at that path, starting with automated sync disabled. 4. Compare the live cluster state against Git until the diff is clean, then enable automated sync and self-heal. ## Results The change people felt first was speed. Creating a new service went from a multi-day, ticket-driven sequence to a form that produces a working repo and a service running in the dev cluster off a single merge. The first deploy happens with no platform-team involvement, which is the whole point. Read and write access to what runs is governed by RBAC and policy, so faster did not mean looser. Adoption is the result that tells you the platform actually worked, and it climbed quickly once the golden path was clearly easier than the old dirt tracks. Within the first quarter the large majority of new services came through templates rather than by hand. The platform team's own time shifted from doing one-off deploys to improving templates and policy, which compounds: one improvement to a template lands in every service scaffolded after it. These numbers are specific to this rollout and were measured on our own usage; treat them as a shape to expect, not a guarantee. The other measurable win was consistency. Because every service starts from the same skeleton, the drift between projects that used to make debugging miserable mostly disappeared. When we needed to roll out a change like a new required label or a security default, we updated the template and the policy, and the fleet converged instead of needing a hand-edit per repo. ## Lessons The most important lesson is that a platform lives or dies by adoption, and adoption is earned by making the paved road genuinely faster than going around it. We resisted the urge to mandate the platform early. Instead we made the golden path the path of least resistance, and teams chose it. A mandate on a platform people dislike just produces malicious compliance. Guardrails have to be automatic to matter. The first time we let a "quick manual approval" creep into a deploy path, we had reinvented the ticket queue in miniature. Encoding the rule as admission policy, so the platform enforces it without a human, is what kept self-service actually self-service. Finally, treat the golden path as a product with a small number of well-maintained templates, not a template for every conceivable variation. A handful of paths that cover the common cases well beats a sprawling catalog nobody trusts. Teams with unusual needs step off the road and still pass policy, and that is fine. The goal was never to control every service, only to make the right thing the easy thing. ## Frequently Asked Questions > **What exactly is a golden path?** A golden path is the supported, opinionated way to do a common task, like creating a new service, with the boring correct defaults already wired in. In this platform it is a Backstage software template that scaffolds a repo, CI, a Helm chart, and the GitOps config in one step. It is a paved road you are free to leave, not a wall you cannot cross. > **Why put Git in the middle instead of deploying straight from Backstage?** Because Git turns auditability and rollback into a solved problem. Every deploy is a reviewable diff with history, and undoing a bad change is a revert that Argo CD reconciles automatically. If Backstage pushed changes directly to clusters, you would lose that trail and have to build approval and rollback yourself. > **How do guardrails avoid becoming the ticket queue you replaced?** They are enforced by machines, not people. RBAC decides who can act, and admission policy with OPA or Kyverno decides what is allowed to run, both automatically at deploy time. There is no human in the routine path clicking approve, which is exactly the bottleneck a manual gate would recreate. > **What happens to teams with genuinely unusual requirements?** They step off the golden path. The platform does not block a team from writing their own manifests or CI, as long as the result still passes policy at admission. The golden path is the default that covers the common cases well, not a hard requirement for every service. > **How do you onboard services that existed before the platform?** Add a catalog-info.yaml so Backstage indexes the service, move its manifests into the config repo, and add an Argo CD Application with automated sync off at first. Once the live state matches Git with a clean diff, turn on automated sync and self-heal. It is an adoption runbook, not a rebuild. > **What is the single most useful metric for this kind of platform?** Adoption, specifically the share of new services created through the golden path rather than by hand. A platform nobody uses is just more software to operate. If teams choose the paved road on their own, it means the road is genuinely faster and safer than the alternative, which is the whole point. > **Do you need Kubernetes to build an internal developer platform?** No. Backstage and GitOps patterns apply to plenty of deployment targets. Kubernetes happens to pair well with Argo CD's reconcile loop and with admission policy, which is why this platform uses it, but the core idea of self-service scaffolding into Git as the source of truth is portable. ## References - [Backstage: Software Templates](https://backstage.io/docs/features/software-templates/) - [Backstage: Software Catalog](https://backstage.io/docs/features/software-catalog/) - [Argo CD: Declarative GitOps CD for Kubernetes](https://argo-cd.readthedocs.io/en/stable/) - [Argo CD: Application resource specification](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/) - [Kyverno: Kubernetes-native policy management](https://kyverno.io/docs/) - [Open Policy Agent (OPA)](https://www.openpolicyagent.org/docs/latest/) - [Team Topologies: platform as a product](https://teamtopologies.com/key-concepts)