--- title: "Node.js Graceful Shutdown: Handle SIGTERM in Express and Fastify" description: "A Node.js snippet for implementing graceful shutdown in Express and Fastify servers. Covers catching SIGTERM/SIGINT, draining in-flight requests, closing database connections, and signaling Kubernetes readiness probe changes during termination." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/codesnippets/post/nodejs-graceful-shutdown-sigterm-express-fastify --- # Node.js Graceful Shutdown: Handle SIGTERM in Express and Fastify **Quick Tip** Node's default SIGTERM handler exits immediately, so every deploy kills your in-flight requests. Here's a shutdown handler that drains them properly, in about 80 lines. ## The Problem ### Requests die mid-response and nothing records it **The Problem** By default, a Node process receiving SIGTERM exits. Right then. Any request currently being handled is cut off mid-response, any transaction in progress is abandoned, and any message pulled off a queue but not yet acknowledged goes back on it or vanishes depending on your broker. The insidious part is the observability gap. Your process is gone before it can log an error, so your error rate looks clean. The client sees `ECONNRESET`, which your APM records as a network blip if it records it at all. You get a small, unexplained error spike during every deploy that nobody ever tracks down because it never points at code. There's a second problem hiding behind the first. Kubernetes sends SIGTERM and starts removing your pod from the Service endpoints at the same moment, and it does not wait for one to finish before doing the other. Endpoint removal has to propagate to kube-proxy on every node, which takes anywhere from a few hundred milliseconds to several seconds on a large cluster. During that window new connections are still being routed to you. _What Kubernetes actually does when a pod terminates_ ### What this costs in practice On a service doing 500 requests per second with 6 pods, a rolling deploy replaces one pod at a time. Each replacement drops whatever that pod was handling. That's a few dozen failed requests per deploy, and if you deploy 20 times a day it's hundreds of user-visible failures a day that never appear on a dashboard. The worse version involves data. A payment handler killed between charging a card and writing the record leaves you with a charge and no order. A queue consumer killed after processing but before acknowledging causes the message to be redelivered, and if the handler isn't idempotent it happens twice. ## The Solution ### Refuse new work, finish existing work, then leave **The Fix** Handle the signal yourself and run an ordered shutdown: stop telling the load balancer you're ready, wait for it to notice, stop accepting new connections while continuing to serve open ones, then close your outbound resources and exit. ### Why the sleep isn't a hack It looks like one. It isn't. The readiness probe failing and the endpoint list updating are two different events separated by real time, and there's no callback that tells you the second has finished. Sleeping is how you wait for a distributed system to converge on something it has already decided. Five seconds is a reasonable default. If your readiness probe runs every 10 seconds you need at least that, plus propagation, so 15 is safer on a big cluster. ## Script Implementation ### Setup and configuration ```ts title="graceful_shutdown.ts" showLineNumbers interface ShutdownOptions { /** How long to keep serving after SIGTERM, waiting for endpoint * removal to reach kube-proxy on every node. Not optional. */ drainDelayMs: number; /** Hard cap on waiting for in-flight requests. Must leave headroom * inside terminationGracePeriodSeconds. */ shutdownTimeoutMs: number; } const options: ShutdownOptions = { drainDelayMs: Number(process.env.DRAIN_DELAY_MS ?? 5_000), shutdownTimeoutMs: Number(process.env.SHUTDOWN_TIMEOUT_MS ?? 15_000), }; // The readiness probe reads this. Flipping it to false is the first // thing shutdown does, and it is what takes us out of rotation. let isShuttingDown = false; ``` ### The cleanup registry ```ts type Cleanup = {name: string; close: () => Promise}; const cleanups: Cleanup[] = []; /** Register anything holding a socket or a file handle. Order matters: * cleanups run in reverse registration order, so register the database * before the things that depend on it. */ cleanups.push({name, close}); } async function runCleanups(): Promise { // Reverse order, and each one isolated: one failing pool should not // prevent the others from closing. for (const {name, close} of [...cleanups].reverse()) { try { await close(); console.log(`[shutdown] closed ${name}`); } catch (err) { console.error(`[shutdown] failed to close ${name}`, err); } } } ``` ### Closing the HTTP server, including keep-alive sockets ```ts const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); function closeServer(server: Server, timeoutMs: number): Promise { return new Promise((resolve) => { let settled = false; const done = () => { if (!settled) { settled = true; resolve(); } }; // Stops accepting new connections. Existing ones keep being served, // and the callback fires once they are all done. server.close(done); // The part people miss. An idle keep-alive socket is an open // connection, so server.close() waits for it even though no request // is in flight. Node 18.2+ lets us drop those immediately. server.closeIdleConnections?.(); // Last resort. If a request is genuinely stuck, we would rather // cut it than get SIGKILLed and cut everything. setTimeout(() => { console.warn('[shutdown] drain timed out, forcing connections closed'); server.closeAllConnections?.(); done(); }, timeoutMs).unref(); }); } ``` ### The signal handler ```ts const shutdown = async (signal: string) => { if (isShuttingDown) { console.log(`[shutdown] already shutting down, ignoring ${signal}`); return; } isShuttingDown = true; console.log(`[shutdown] ${signal} received`); // 1. Fail readiness immediately. This starts the clock on // Kubernetes taking us out of the endpoint list. // 2. Keep serving while that propagates. await sleep(options.drainDelayMs); // 3. Stop accepting, finish what is open. await closeServer(server, options.shutdownTimeoutMs); // 4. Now nothing is using the database, so close it. await runCleanups(); console.log('[shutdown] done'); process.exit(0); }; process.on('SIGTERM', () => void shutdown('SIGTERM')); process.on('SIGINT', () => void shutdown('SIGINT')); // An unhandled rejection leaves the process in an unknown state. // Shut down rather than continuing to serve from it. process.on('unhandledRejection', (reason) => { console.error('[shutdown] unhandled rejection', reason); void shutdown('unhandledRejection'); }); } ``` ### Wiring it into Express ```ts installShutdownHandlers, isReady, onShutdown, } from './graceful_shutdown'; const app = express(); const pool = new Pool({connectionString: process.env.DATABASE_URL}); // Readiness reflects shutdown state. Liveness must NOT: a shutting-down // process is unhealthy for traffic but perfectly alive, and failing // liveness here would make the kubelet kill it mid-drain. app.get('/readyz', (_req, res) => isReady() ? res.sendStatus(200) : res.sendStatus(503), ); app.get('/livez', (_req, res) => res.sendStatus(200)); app.get('/', async (_req, res) => { const {rows} = await pool.query('select now()'); res.json(rows[0]); }); const server = app.listen(3000, () => console.log('listening on 3000')); // Registered first, so it closes last, after the server has drained. onShutdown('postgres pool', () => pool.end()); installShutdownHandlers(server); ``` ### Wiring it into Fastify ```ts const app = Fastify({logger: true}); const pool = new Pool({connectionString: process.env.DATABASE_URL}); app.get('/readyz', async (_req, reply) => isReady() ? reply.code(200).send() : reply.code(503).send(), ); app.addHook('onClose', async () => { await pool.end(); }); await app.listen({port: 3000, host: '0.0.0.0'}); // Fastify's own close() already drains and runs onClose hooks, so the // only thing missing is the delay before it starts. const shutdown = async (signal: string) => { app.log.info(`${signal} received`); await new Promise((r) => setTimeout(r, 5_000)); // endpoint propagation await app.close(); process.exit(0); }; process.on('SIGTERM', () => void shutdown('SIGTERM')); process.on('SIGINT', () => void shutdown('SIGINT')); ``` ## Usage and Benefits ### The Kubernetes side has to agree **Why This Helps** The handler above is only half of it. The pod spec has to give it enough time, and the numbers have to add up. ```yaml spec: # 5s drain delay + 15s shutdown timeout + pool close, plus headroom. # If this is smaller than the sum, you get SIGKILLed mid-drain and # lose the requests you were protecting. terminationGracePeriodSeconds: 45 containers: - name: api readinessProbe: httpGet: {path: /readyz, port: 3000} periodSeconds: 5 livenessProbe: httpGet: {path: /livez, port: 3000} periodSeconds: 10 ``` Two things that quietly break this. If your container runs under `npm start` or a shell wrapper, PID 1 is the shell, and it does not forward SIGTERM to Node. Your handler never runs. Either exec the binary directly in `CMD`, or add an init like `tini`. And a `readinessProbe` with a `periodSeconds` of 30 means up to 30 seconds before Kubernetes even notices you failed it, which is longer than your drain delay. Keep it at 5. ### Verify it actually works ```bash # Locally: watch a long request survive the signal curl -s localhost:3000/slow & sleep 0.5 kill -TERM $(pgrep -f 'node dist/index.js') # The curl should complete. The logs should show the ordered shutdown. # In a cluster: hammer it while deleting the pod kubectl run load --rm -it --image=williamyeh/hey -- \ -z 60s -c 20 http://api.default.svc.cluster.local/ kubectl delete pod api-xxxx # The success rate should stay at 100%. Any non-2xx is a dropped # request, and the usual cause is a drain delay that is too short. ``` That second test is the only one that matters, because it exercises the endpoint propagation timing that you cannot reproduce locally. Run it before you believe any of this works. ## Community Discussion ### How do you handle this? **Your Turn** I'd like to know what drain delay you settled on and how you arrived at it. Five seconds is my default and I have no strong evidence for it beyond "it stopped the errors." If you measured your actual endpoint propagation time, I want to hear the number and how you got it. The other thing I'm curious about is queue consumers. HTTP draining is well understood, but a worker holding a long-running job when SIGTERM arrives is a harder problem. Do you finish the job, requeue it, or checkpoint it? ### Alternative approaches There are lighter options than writing this yourself. `terminus` wraps the same logic with a nicer API and handles the probe endpoints for you. `stoppable` patches the server to track connections and close idle ones, which mattered more before `closeIdleConnections()` existed in core. If you run a service mesh, some of this moves out of your process. You can configure Envoy to drain on its own, though you still need the sleep, because the mesh sidecar has its own shutdown ordering problem: if the proxy exits before your app finishes, your outbound calls start failing mid-drain. And a `preStop` hook running `sleep 5` moves the delay into the pod spec instead of your code. It works, it's language-agnostic, and I don't love it, because the delay is now invisible to anyone reading the application. Same result, worse discoverability.