Graceful Shutdown: Stop Accepting, Drain, Then Exit

The idea: a deploy should not drop a single request

Every deploy, scale-down or node drain stops old server processes. The stopper (the kubelet, systemd, docker stop, an autoscaler) sends SIGTERM first and SIGKILL later. The time in between is the grace period. A graceful shutdown uses that time in three moves: stop getting new work, finish the work you have (drain), then exit 0.

The canvas shows clients, a load balancer and two servers. Server A is the one being replaced; B stays up. Clients send one request per second, and each takes 3 s. The load balancer sends each request to the next backend in its pool (round robin). Every 2 s it sends GET /healthz to each backend, and two failed checks in a row take a backend out. Each server shows its listen socket (fd 3, port 8080), its health answer and its in-flight requests, one connection fd each. The timeline records A second by second.

Exit at once: in-flight requests are cut

A process with no SIGTERM handler dies on the default action (exit code 143 = 128 + 15). The kernel closes every socket. The requests in flight get a TCP RST, and the load balancer turns each into a 502 Bad Gateway. Worse, the load balancer still believes A is healthy, so it keeps sending it new requests. They hit a closed port and are refused, until the health checks catch up (Demo: exit at once: r1 is cut, r3 and r5 are refused, 3 dropped).

Closing the listener is not enough

Most frameworks ship a "graceful" stop that closes the listening socket and waits for in-flight requests: server.close() in Node.js, http.Server.Shutdown(ctx) in Go. In-flight work is now safe. But nothing told the load balancer. For up to interval × unhealthy threshold (2 s × 2 = 4 s here) it keeps sending new connections to a port that nobody listens on (Demo: close the listener only: r1 finishes, but r3 and r5 get ECONNREFUSED, 2 dropped).

The readiness flip: tell the load balancer first

The fix is to change the order. On SIGTERM the server keeps its listener open and keeps serving, but its readiness endpoint starts to answer 503. The load balancer sees the failed checks and takes the server out of the pool. Only then, after a fixed shutdown delay, does the server close the listener and drain (Demo: readiness flip, then drain: out of the pool at t=4, listener closed at t=5, exit 0 at t=6, nothing dropped).

SIGTERM handler	At SIGTERM	A out of pool	A exits	Dropped
none (exit at once)	process dies	t=4	t=0, code 143	3
close listener only	fd 3 closed	t=4	t=2, code 0	2
readiness flip + drain	/healthz → 503	t=4	t=6, code 0	0

Pick the delay from the load balancer's settings: at least interval × unhealthy threshold, plus a margin for slow config pushes. In Kubernetes the "load balancer" is the EndpointSlice and every kube-proxy and Ingress that watches it; there the pod is marked terminating as soon as it is deleted, and the delay covers how late the routers hear about it. That is the same reason people add a preStop sleep.

Keep-alive connections

Load balancers and HTTP clients keep connections open and reuse them (keep-alive). Closing the listener does not touch them, so two things go wrong: new requests can still arrive on an old connection, and the process never sees "no connections left" and does not exit. A draining server answers with Connection: close (HTTP/2: a GOAWAY frame), so the client opens its next request elsewhere, and it closes idle keep-alive connections itself. On the canvas, A's replies after SIGTERM carry Connection: close. Go's Shutdown does this for you; in Node.js call server.closeIdleConnections() (or closeAllConnections() at the deadline).

The grace period ends with SIGKILL

Draining has a deadline. If the process is still alive when the grace period ends (terminationGracePeriodSeconds, 30 s by default, 10 s on this page), it gets SIGKILL, which cannot be caught: exit code 137 = 128 + 9, and whatever is still in flight is cut (Demo: slow upload hits SIGKILL: the 12 s upload started at t=0 dies at t=10). Two fixes: raise the grace period above your longest request, or give the drain its own shorter timeout so the app can log, close cleanly and exit before the kill.

// Node.js: flip, wait, drain
let ready = true;
app.get('/healthz', (req, res) => res.status(ready ? 200 : 503).end());
process.on('SIGTERM', () => {
  ready = false;                        // load balancer stops sending
  setTimeout(() => {
    server.close(() => process.exit(0)); // stop accepting, exit when drained
    server.closeIdleConnections();
  }, 5000);                             // shutdown delay
});

Other servers have the same pieces under other names: nginx -s quit (SIGQUIT) is nginx's graceful stop, because nginx treats SIGTERM as "stop now"; Spring Boot has server.shutdown=graceful; Envoy and most cloud load balancers have a connection draining or deregistration delay that keeps old connections alive while new ones go elsewhere. See also Load Balancer: L4 vs L7 for health checks and pools, and Canary and Blue/Green Deployments for moving traffic between versions.

What the page leaves out

One load balancer, two servers and steady traffic are drawn. Load balancer retries (which hide some 502s for idempotent requests), HTTP/2 GOAWAY frames, WebSockets and other long-lived streams, background jobs and queue consumers that must stop pulling work, database connection pools closing, and the Kubernetes EndpointSlice race in detail are not shown. For the pod side of the story see The Life of a Kubernetes Pod.

References

Kubernetes: Pod Lifecycle, Termination of Pods

Node.js: server.close() and closeIdleConnections()

Go: net/http Server.Shutdown

AWS: Health checks for Application Load Balancer target groups