What a Go service owes its readiness probe

Liveness asks "is this process wedged? if so, kill it." Readiness asks "should this pod get traffic right now?" A failed liveness check restarts the pod. A failed readiness check just removes it from the Service's endpoints until it passes again. Only one of those helps when Postgres is slow, and it isn't the restart.

So, the rules I've settled on:

• Liveness checks nothing external. Return 200 if the process can answer an HTTP request. That's the whole check. A liveness probe that pings the database turns a database blip into a fleet-wide restart storm.
• Readiness checks what you need to serve: a database ping with a short timeout, whether the config loaded, whether the cache warmed. Cache the result for a few seconds; the kubelet probes every five to ten seconds, and one database round-trip per probe per pod, times however many pods you run, adds up to a surprising amount of load that does nothing.
• On SIGTERM, fail readiness first. Then wait a few seconds, then stop the listener. Kubernetes removes the pod from endpoints asynchronously; if you close the listener the instant the signal arrives, requests that were routed a moment ago get connection refused.

var ready atomic.Bool

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    mux.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) {
        if !ready.Load() || !depsOK() {
            http.Error(w, "not ready", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}
    go func() { _ = srv.ListenAndServe() }()
    ready.Store(true)

    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
    defer stop()
    <-ctx.Done()

    ready.Store(false)           // 1. fail readiness
    time.Sleep(5 * time.Second)  // 2. let endpoints catch up
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    _ = srv.Shutdown(shutdownCtx) // 3. drain in-flight requests
}

The time.Sleep is not a smell here; it's the endpoint-propagation delay made explicit, and I'd rather it were a named constant than a comment. Make sure terminationGracePeriodSeconds is larger than the sleep plus the shutdown timeout, or the kubelet will send SIGKILL in the middle of the drain and undo the point of all this. atomic.Bool is Go 1.19+; before that, an int32 and atomic.LoadInt32 did the same job less legibly.