Rate limiting across many servers

A global limit, local counters

“100 requests per minute per API key” is one line of code on a single server: keep a counter, check it, reject over the cap. Now put fifty load-balanced servers behind it. A key’s requests scatter across all of them, so no single server sees the whole picture — each holds a fraction of the count. Whose counter decides? This is the real-world rate-limiting problem (distinct from which algorithm — token vs leaky bucket — and from designing a standalone limiter service): it’s about coordinating one logical limit across N machines, and every approach trades accuracy against latency.

The naive split (and why it drifts)

Tempting answer: give each of N servers a local limit of 100 / N. It needs no coordination — and it’s wrong in practice:

• Uneven load — load balancers don’t spread one key’s traffic perfectly; sticky sessions and hot keys make some servers see far more than their share, so they reject early while others sit idle.
• Autoscaling — N changes constantly, so the per-server slice is a moving target.

You end up enforcing something between 100/N and 100·N depending on the skew — not “100.”

Centralized counter

Keep the count in a shared store (typically Redis) that every server reads and updates atomically — an INCR with expiry, or a token-bucket check in a Lua script so the read-modify-write is atomic:

• Accurate — one global view, so the limit means exactly what it says.
• Cost — a network hop on every request (added latency), and the store becomes a hot dependency: a bottleneck and a SPOF. Mitigate by sharding the counter by key across Redis nodes (a given key’s traffic isn’t that high), pipelining, and degrading open/closed if Redis is unreachable.

This is the default for correctness-sensitive limits.

Local + sync (approximate)

Each server enforces against a local counter and periodically reconciles — broadcasting or gossiping its counts so everyone converges on the global total:

• Fast — no per-request network hop; decisions are in-memory.
• Approximate — between syncs, servers don’t know each other’s recent counts, so the aggregate can overshoot the limit (everyone admits traffic before the next reconciliation). Tighter sync = more accurate but chattier.

Good when a little overage is acceptable and latency is precious.

Sticky routing

Route each key consistently to one server (hash the API key → a node), so that node’s local counter is authoritative for that key — accurate and hop-free:

• Cost — a hot key concentrates on one node (back to the hot-shard problem), and scaling/rebalancing reshuffles ownership. Pairs naturally with consistent hashing.

The trade-off

The dial is accuracy vs latency vs availability — pick by how much overage you can tolerate and how tight your latency budget is.

Where it shows up

API gateways (Kong, Envoy, AWS API Gateway) doing per-key limits across a fleet; Stripe-style per-account quotas; abuse protection at the edge — all of which solve the “global limit, many servers” coordination problem under the hood.

The interview cue

When a design enforces a limit and you’ve already got many servers, name the coordination problem: “The limit is global but each server sees only part of the traffic. For accuracy I’d keep the counter in shared Redis with atomic ops, accepting a network hop and sharding the counter to avoid a hotspot. If latency mattered more than precision I’d enforce locally and sync periodically, or route each key to one server so its local counter is authoritative.” Showing you know the single-box solution doesn’t generalize — and naming the accuracy/latency trade — is the point.