Duplicate requests and the idempotency gap

Exactly-once is mostly a fiction

A client sends “charge $50,” the request succeeds, but the response is lost to a timeout. The client has no idea whether it worked, so it retries — and the card is charged twice. The uncomfortable truth of distributed systems: you cannot guarantee a message is delivered exactly once, only at least once (might repeat) or at most once (might vanish). Since silently losing a payment is unacceptable, real systems choose at-least-once — which means every write must be built to survive being delivered twice. The bugs come from operations that aren’t.

Where duplicates come from

They’re not edge cases; they’re routine:

• The ambiguous timeout — the request succeeded but the ack was lost, so the client retries a write that already happened. The single most common source.
• At-least-once queues — a consumer crashes after doing the work but before acking; the broker redelivers and the work runs again.
• Infrastructure retries — a load balancer or gateway re-sends a request it thinks failed.
• User double-submit — an impatient user double-clicks “Pay,” or the back button re-posts the form.
• Replays — reprocessing a stream or replaying a dead-letter queue after a fix.

The gap: not every operation is safe to repeat

An operation is idempotent if doing it twice leaves the same result as doing it once. Some are naturally so; some are landmines:

safe to repeat (idempotent)        dangerous (not idempotent)
─────────────────────────────      ──────────────────────────
GET /order/42                      INSERT a new order row
PUT user.email = "a@b.com"         balance = balance + 50   (increments)
DELETE /session/x                  "charge the card"
set status = SHIPPED               "send the email / push"

Reads, deletes, and “set field to value” land in the same place no matter how many times they run. The danger zone is anything that creates a new row, increments a counter, moves money, or triggers a side effect — repeat it and you get a duplicate order, a doubled balance, two charges, two emails. The idempotency gap is exactly these non-idempotent writes meeting at-least-once delivery.

Closing the gap

The discipline is to assume every write can arrive twice and make it idempotent anyway — not to chase the impossible goal of perfect deduplication upstream. The levers (each detailed elsewhere): attach a client-generated idempotency key and record processed keys so a repeat is a no-op returning the first result; rely on a natural unique constraint (one row per order_id) so the second insert is rejected; or use a conditional write (compare-and-set on a version) so a stale duplicate can’t apply. The point of this lesson is upstream of the technique: spot which operations are in the danger zone and design them to tolerate replay.

Where it shows up

Payment APIs (Stripe’s Idempotency-Key header is the canonical example), order placement, “create resource” endpoints, webhook receivers (providers retry until they get a 2xx), and every at-least-once message consumer.

The interview cue

When a design has a write that matters — a payment, an order, a notification — flag it: “Delivery is at-least-once and clients retry on timeouts, so this write will arrive twice. It’s not naturally idempotent — charging a card twice is a real bug — so I’d make it idempotent with a client-supplied idempotency key (or a unique constraint) that turns the duplicate into a safe no-op.” Identifying which operations are unsafe to repeat, and treating duplicates as the default rather than the exception, is the maturity signal.