Kubernetes Health Probes: Building Self-Healing Applications

Kubernetes has a reputation for keeping applications online, but it is not magic. Out of the box, the cluster is blind to what happens inside your container. If your application deadlocks or exhausts its database connection pool, the process is often still running. To Kubernetes, a running process looks like a healthy pod, so it keeps forwarding traffic to an endpoint that stopped responding minutes ago.

To get a genuinely self-healing application, you have to translate your app’s internal state into signals the control plane understands. That is what health probes are for. Configure them well and the cluster routes traffic away from overwhelmed pods and restarts frozen ones on its own. Configure them badly and you hand yourself a new failure mode: cascading restarts that take down the whole service at once.

What self-healing actually means

Self-healing is the cluster automatically detecting, isolating, and replacing unhealthy workload instances with no human in the loop. Pods and containers are meant to be disposable. Instead of nursing one fragile server, Kubernetes uses higher-level controllers (Deployments, StatefulSets) to constantly compare the current state of the cluster against the state you declared.

The local node agent, the kubelet, runs a continuous polling loop to track container health. If a container process crashes and exits with a non-zero code, the kubelet notices right away and restarts it per the pod’s restartPolicy.

The harder problem is that modern systems fail quietly. A background thread panics, a memory leak triggers endless garbage-collection pauses, an upstream API stops answering. The process never exits, so the kubelet sees nothing wrong. Probes are how you give the kubelet the diagnostics to catch those invisible failures and act, either by pulling the pod out of rotation or by terminating and replacing it.

The three probes, and what each one controls

Kubernetes gives you three probe types. Each answers a different question and triggers a very different action on failure. Mixing up their jobs is a leading cause of cluster instability.

When a pod launches, the kubelet checks whether a startup probe is defined. If it is, liveness and readiness checks are paused until the startup probe succeeds, which gives your code time to initialize. Once the startup probe passes, the kubelet begins running the liveness and readiness probes periodically.

How a probe failure turns into an action

The two failure paths are handled by different parts of the cluster, and the difference matters.

When a liveness probe fails failureThreshold times in a row, the kubelet treats the container as unhealthy and starts terminating it: it sends SIGTERM, waits out terminationGracePeriodSeconds, and issues SIGKILL if the container refuses to exit. Then it restarts the container behind an exponential backoff that starts at 10 seconds and grows up to 300 seconds, so a fast crash loop cannot melt the node’s CPU.

When a readiness probe fails, nothing gets killed. The kubelet sets the pod’s Ready condition to False. The Endpoints controller watches those conditions, and the moment a pod goes unready it removes the pod’s IP from the EndpointSlice for that Service. kube-proxy on each node picks up the update and rewrites the local network rules, so new traffic stops going to the struggling pod. Established TCP connections usually stay put; new requests route cleanly to healthy replicas.

That is the whole point of keeping the two separate: readiness is a reversible “step out of the line for a moment,” liveness is a final “you are broken, start over.”

The misconfigurations that cause outages

Probes are simple, but a couple of architectural misunderstandings turn them into self-inflicted incidents.

Checking your database in a liveness probe

This is the most dangerous probe mistake there is. The tempting move is to point both liveness and readiness at one generic /health endpoint that checks the database, the Redis cache, and a couple of third-party APIs.

Picture a service that talks to PostgreSQL. The database fails over to a new primary, and queries fail for about 30 seconds. If your liveness probe pings the database, liveness fails, and the kubelet kills the container. Every pod loses database connectivity at the same instant, so Kubernetes restarts your entire fleet at the same instant. A 30-second blip becomes a full cascading outage, even though your application runtime was perfectly fine. When the database comes back, it is immediately hit by a thundering herd of restarting pods each opening fresh connection pools, which frequently knocks the database over again.

Thresholds and timeouts that are too aggressive

timeoutSeconds is how long the kubelet waits for the probe handler to answer. Under CPU pressure an app might take two or three seconds to respond to a health check. If timeoutSeconds is 1 and failureThreshold is low, a single slow response or one dropped packet evicts and restarts the pod. Give probes enough headroom to survive a busy moment.

Writing the probe handlers

Kubernetes supports four handler mechanisms. You configure them inside spec.containers.

HTTP is the common choice for web services. The kubelet sends a GET and treats any status from 200 to 399 as success.

1
livenessProbe:
2
  httpGet:
3
    path: /healthz/live
4
    port: 8080
5
    httpHeaders:
6
      - name: X-Custom-Auth
7
        value: internal-probe
8
  periodSeconds: 15
9
  failureThreshold: 3

TCP socket suits non-HTTP apps. The kubelet just opens a connection on the port; if the socket opens, the probe passes.

1
readinessProbe:
2
  tcpSocket:
3
    port: 5432
4
  initialDelaySeconds: 5
5
  periodSeconds: 10

Exec runs a command inside the container and passes on exit code 0. Handy for apps that write a status file, but it is the most expensive option because the runtime spawns a process for every check.

1
readinessProbe:
2
  exec:
3
    command:
4
      - cat
5
      - /tmp/healthy
6
  periodSeconds: 5

gRPC used to require bundling a separate health-check binary and calling it through an exec probe. Kubernetes now speaks the standard gRPC Health Checking Protocol natively.

1
livenessProbe:
2
  grpc:
3
    port: 50051
4
  periodSeconds: 10

When you actually need a startup probe

Before startup probes existed, health-checking apps with unpredictable boot times was painful. A legacy Spring Boot monolith might download config, run database migrations, and warm caches before it can serve anything.

The old workaround was a large initialDelaySeconds so liveness would not kill the app mid-boot. That created a nasty blind spot: if the app booted fine in 30 seconds but deadlocked two minutes later, Kubernetes would not notice until the whole padded initial delay elapsed. The startup probe fixes this by separating “still booting” from steady-state health, so liveness can stay tight without risking a premature kill during a slow start.

Zero-downtime rollouts

True zero-downtime deploys need Kubernetes internal routing to stay in sync with external infrastructure, and that is exactly where naive rollouts drop requests, especially behind a cloud load balancer like an AWS ALB.

The load balancer registration gap

During a rolling update the Deployment controller brings up a new pod. As soon as its readiness probe returns success, Kubernetes marks the pod Ready, adds it to the Service endpoints, and sends SIGTERM to an old pod to scale down. But the external load balancer runs on its own schedule: the target group may take another 10 to 15 seconds to register the new IP, run its own checks, and start routing. In that window the old pod is going away and the new pod is not receiving traffic yet, so users get dropped requests and 502s.

Readiness gates close the gap

A readiness gate says the pod cannot be marked Ready until an external controller confirms it, even after every container readiness probe passes. The ALB controller flips that condition once target registration is done.

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: payment-service
5
spec:
6
  template:
7
    spec:
8
      readinessGates:
9
        # The ALB controller flips this once target registration completes
10
        - conditionType: 'target-health.alb.k8s.aws'
11
      containers:
12
        - name: app
13
          image: payments:v2.1
14
          readinessProbe:
15
            httpGet:
16
              path: /actuator/health/readiness
17
              port: 8080

With the gate in place, Kubernetes will not terminate the old pods until the load balancer signals that the new pods are actually taking traffic.

The preStop hook and draining

At the end of a pod’s life the kubelet removes its IP from the endpoints and sends SIGTERM at the same time. Because endpoint removal takes a moment to propagate through iptables rules and external load balancers, in-flight requests can still arrive for a few seconds after shutdown begins. A preStop hook runs synchronously before SIGTERM, so a short sleep pauses shutdown long enough for the removal to propagate everywhere.

1
lifecycle:
2
  preStop:
3
    exec:
4
      command: ['/bin/sh', '-c', 'sleep 15']

Frequently asked questions

References

• Pod lifecycle
• Configure liveness, readiness and startup probes
• Pod conditions and readiness gates
• Container lifecycle hooks
• gRPC probes now in beta
• Liveness and readiness probes with Spring Boot
• Kubernetes liveness probes are dangerous
• Zero-downtime deployments on AWS EKS