Containing cascading failures

How one slow service takes down many

Service D gets slow — not down, just slow. Service C calls D and now its threads sit blocked waiting. Requests pile up, C’s thread pool fills, so C goes slow too. B calls C and blocks; B fills up. Soon the entire chain up to the user is unresponsive, because of a single sluggish dependency at the bottom. Nothing crashed — they all ran out of the resource (threads, connections, memory) needed to serve anyone, while waiting on something that never answered. That’s a cascading failure: localized trouble propagating along the call graph until the system is down.

The mechanism: resource exhaustion

The carrier of the cascade is almost always a bounded resource held while waiting: a thread blocked on a call, a connection checked out of the pool, a request queued in memory. A slow dependency doesn’t fail your service directly — it ties up these resources until there are none left for healthy work, and the waiting is what kills you. So containment is mostly about refusing to wait unboundedly and isolating who can consume what.

The containment toolkit

• Timeouts — never wait forever. Every remote call gets a deadline so a hung dependency releases the thread instead of holding it indefinitely. This is the single most important one.
• Circuit breakers — track a dependency’s failure rate; when it crosses a threshold, open the circuit and fail fast for a cooldown instead of calling the dead service at all, then probe to see if it recovered. Stops you from pouring requests (and retries) into a black hole.
• Bulkheads — isolate resource pools so one dependency can’t drain everything. Give calls to D their own bounded thread/connection pool; when D slows, only that pool exhausts and the rest of the service keeps serving. (Named after a ship’s watertight compartments.)
• Load shedding — when overloaded, reject excess work early (return 429) to protect the core rather than accepting everything and collapsing. Shedding 10% keeps the other 90% healthy.
• Backpressure — push the “slow down” signal upstream (bounded queues that block or reject when full) so producers throttle instead of piling unbounded work on a struggling consumer.
• Graceful degradation — when a dependency is unavailable, serve a degraded answer (cached/stale data, a default, a disabled non-critical feature) instead of failing the whole request. The feed loads without the “people you may know” widget rather than not at all.

caller ──timeout──> [bulkhead pool for D] ──circuit breaker──> D
   │                                                            (open → fail fast)
   └─ overloaded? shed load (429) ; queue full? backpressure upstream

Where it shows up

Netflix Hystrix popularized circuit breakers and bulkheads; resilience4j, Envoy, and Istio bake timeouts, circuit breaking, and outlier ejection into the mesh. Every mature microservice architecture combines these — they’re the difference between “one service degraded” and “the site is down.”

The interview cue

When your design has services calling services, pre-empt the blast radius: “To keep one slow dependency from cascading, I’d put a timeout on every call, a circuit breaker to fail fast when it’s down, bulkhead its thread pool so it can’t starve the rest, shed load under overload, and degrade gracefully — serve stale or drop the non-critical widget rather than failing the whole page.” Listing the toolkit and naming resource exhaustion as the mechanism is the senior answer.