Retry storms and how to tame them

Retries are good engineering until a blip makes every client retry at once — multiplying load and turning a brief slowdown into a full outage you inflicted on yourself.

When retrying makes it worse

A service hiccups for a second. Every client times out and retries. Now the already-struggling service gets 2–3× its normal load at the worst possible moment, slows further, causes more timeouts, which trigger more retries — a feedback loop that drives a momentary brown-out into a sustained outage. The retry logic meant to improve reliability has become a self-inflicted DDoS. This is a retry storm, and at scale it’s one of the most common ways systems fall over.

How the storm builds

Synchronized retries — many clients failed at the same instant and all retry on the same fixed delay, arriving together as a thundering herd.
Work amplification — each user request becomes several attempts; multiply that across every layer (client → gateway → service → database, each retrying) and one request can fan out into dozens of calls.
Retries on overload — the cruel part: the service is failing because it’s overloaded, and retries add exactly the load that’s killing it.

Backoff and jitter

Two cheap mechanisms defuse most of it:

Exponential backoff — wait longer after each failure (1s, 2s, 4s, 8s…) so a struggling service gets breathing room instead of an immediate repeat.
Jitter — randomize the delay so retries desynchronize. Without jitter, every client backs off by the same amount and the herd just regroups and hits again in sync. Full jitter spreads them across the interval:

delay = random(0, min(cap, base * 2^attempt))   # full jitter

Jitter is the part people omit — and it’s what actually breaks the synchronized herd.

Caps, budgets, and circuit breakers

Backoff slows retries; you also need to bound them:

A retry cap — a small maximum number of attempts (2–3), never unbounded.
A retry budget — allow retries only up to, say, 10% of total requests; once the failure rate blows the budget, stop retrying and fail fast. This prevents the multiplier from ever forming.
Circuit breakers — when a dependency is clearly down, stop calling it entirely for a cooldown and fail immediately, so you’re not retrying a corpse (its own resilience pattern, next).
Only retry the retryable — retry transient errors (timeout, 503), never a deterministic 400 that will fail identically every time.

Where it shows up

Built into mature clients and SDKs: AWS SDK retries with exponential backoff and jitter, gRPC retry policies with budgets, Envoy/Istio retry caps, and resilience libraries (Polly, resilience4j). Any design with a client calling a remote service should specify the retry policy, not just “we retry.”

The interview cue

When you add retries, immediately bound them: “I’d retry only transient failures, with exponential backoff and jitter so clients don’t resynchronize into a thundering herd, plus a small attempt cap and a retry budget — and a circuit breaker so we stop hammering a dependency that’s already down.” Showing that you know retries amplify load, and naming jitter and budgets, separates you from “just add a retry.”