Rebalancing shards when traffic is skewed

Balanced by design isn’t balanced in practice

You picked a high-cardinality shard key and the data sits evenly across nodes. Yet one shard is melting while the others idle. That’s because access is skewed even when storage isn’t. Real workloads follow a Zipf curve: a few keys get most of the traffic. A celebrity’s profile, a viral video, your biggest enterprise tenant, or simply “everything written today” all land on one shard and overwhelm it. Consistent hashing minimizes movement when nodes change — but it does nothing about a single key that’s red-hot. This is a different problem: detecting and relieving a hot shard while serving live traffic.

How a shard goes hot

• A hot key — one celebrity row, one popular product, one tenant 100× the others. Hashing scatters keys evenly, but a single key still lives on a single shard, and that key alone can saturate it.
• A hot range — with range partitioning, monotonic keys (timestamps, auto-increment ids) pile all new writes onto the last shard.
• Drift over time — a shard map that was balanced at launch skews as some tenants grow and others churn.

Detecting it

You can’t fix what you can’t see. Instrument per-shard (and ideally per-key) metrics — requests/sec, CPU, p99 latency, bytes — not just cluster-wide averages, which hide the imbalance. Track the top-K keys per shard (a heavy-hitters sketch like count-min) so you can tell a uniformly busy shard (needs more capacity) from one dominated by a single hot key (needs a different fix).

Relieving it without downtime

Split a hot range. Pick a split point that halves the load (not just the key space) and hand half to a new shard. The live migration is the careful part:

1. start copying the range to the new shard in the background
2. dual-write new mutations to both old and new shard
3. once caught up, flip the routing/directory entry to the new shard
4. stop dual-writing; drop the moved range from the old shard

A directory-based map (key → shard lookup) makes step 3 a single atomic pointer flip, which is why elastic stores keep one. A hot single key can’t be split by range — instead spread it: append a random suffix to fan one key across N sub-keys (and scatter-gather on read), or front it with a cache so most reads never reach the shard at all.

Where it shows up

• HBase / Bigtable auto-split regions when they grow or heat up.
• DynamoDB adaptive capacity isolates and even splits a partition “for heat.”
• Vitess / Citus resharding splits and migrates ranges online.
• Kafka partition skew from a bad key forces a repartition.

The interview cue

When you propose sharding, pre-empt the follow-up: “I’d shard by user id, but I’d watch per-shard load because traffic is Zipfian — a celebrity or a whale tenant will hot-spot one shard. I’d detect it with per-shard and top-K-key metrics, then either split the range and migrate it live behind a directory flip, or, for a single hot key, cache it and fan it across sub-keys.” Naming detection plus live resplit, not just “consistent hashing,” is the senior signal.