← Articles

Graceful Shutdown in Go Services

· 2 min read
Go Distributed Systems

Most Go shutdown code looks like this, and most of it is subtly wrong:

sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
<-sig

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
srv.Shutdown(ctx)

It stops accepting new connections and waits for in-flight requests. That’s the easy half.

The half that gets skipped

The moment you receive SIGTERM, your load balancer usually doesn’t know yet. Its health check runs on an interval — maybe every 5 seconds, maybe every 30 — so for that window it is still routing new requests to a process that just stopped listening. Those become 502s during every single deploy.

The fix is to fail readiness before you stop serving:

  1. Receive SIGTERM.
  2. Flip the readiness endpoint to unhealthy. Keep serving normally.
  3. Sleep for longer than the load balancer’s check interval × threshold.
  4. Now call srv.Shutdown(ctx).
<-sig
ready.Store(false)
time.Sleep(shutdownDelay) // > LB detection window
srv.Shutdown(ctx)

Background work needs its own budget

srv.Shutdown only knows about HTTP. Consumers, cron loops and worker pools have to be drained separately — and the sum of everything must fit inside your orchestrator’s grace period, or it will SIGKILL you mid-drain. On Kubernetes that’s terminationGracePeriodSeconds, 30 by default.

Budget it explicitly:

  • LB detection delay: 5s
  • HTTP drain: 10s
  • Worker drain: 10s
  • Headroom: 5s

Thirty seconds exactly. If your numbers don’t add up to something under the grace period, the graceful path is decoration — the kernel is what’s actually ending your process.

Test it

Send yourself a SIGTERM under load and count the failed requests. If the number isn’t zero, one of the steps above is missing. This is the only test that matters, and it’s the one nobody writes.