Skip to content
System design course
Ch.2 · The building blocks·concept ·6 min read

Poison messages and dead-letter queues

One message that can never succeed will be redelivered forever, burning capacity or blocking the queue — the dead-letter queue is the escape hatch.

The message that never succeeds

Queues deliver at least once: if a consumer doesn’t acknowledge, the message comes back. That’s exactly what you want for transient failures — but it’s a trap for a poison message, one that fails every time. The consumer picks it up, crashes or throws, doesn’t ack, the broker redelivers it, and you’re in an infinite loop: the same message processed forever, wasting capacity — or worse, in an ordered stream, stuck at the head blocking everything behind it.

Why messages go poison

  • Malformed payload — truncated, wrong schema, unparseable JSON.
  • A consumer bug — a null field hits a code path that always throws.
  • A missing dependency — references a row that was deleted, or a downstream that’s permanently gone.
  • Schema drift — the producer upgraded its format and old consumers can’t read it.

The common thread: retrying won’t help, because nothing about the situation will change on the next attempt.

The dead-letter queue

Give up after a bounded number of tries and set the message aside instead of looping. The broker tracks a delivery/receive count; once it crosses a threshold, the message is moved to a separate dead-letter queue (DLQ):

receive message -> process
  success            -> ack (done)
  failure, count < N -> nack, redeliver later (transient retry)
  failure, count = N -> move to DLQ, ack original (stop blocking the main queue)

This keeps the main queue flowing — poison messages step aside so healthy traffic isn’t starved — while preserving the bad messages for later instead of dropping them.

Operating the DLQ

A DLQ is only useful if someone looks at it. Alert on DLQ depth > 0 (a non-empty DLQ is a bug signal). Attach metadata — the failure reason, attempt count, original timestamp — so triage is possible. After you ship a fix, redrive: replay the DLQ back onto the main queue. And mind head-of-line blocking: in a strictly ordered partition (Kafka), you can’t just skip the poison message without breaking order, so you either park it and advance the offset deliberately or accept that the partition stalls until it’s handled.

Where it shows up

SQS redrive policy (maxReceiveCount → DLQ), RabbitMQ dead-letter exchanges, Kafka with retry topics and a dead-letter topic, and effectively every managed queue. Any at-least-once pipeline needs one.

The interview cue

Whenever your design has a queue or consumer, add the failure path unprompted: “Delivery is at-least-once, so I’d cap retries and route messages that fail N times to a dead-letter queue — that stops a poison message from looping forever or blocking the partition, and lets us alert, inspect, and redrive after a fix.” Calling out poison messages and the DLQ shows you’ve operated a queue, not just drawn one.