Handling events that arrive out of order

The order you send isn’t the order you get

Producer emits create account, then deposit $100. The consumer sees deposit first and rejects it — no such account. Or two updates to the same row apply in the wrong order and a stale value clobbers the fresh one. In a single process, events are naturally ordered; the moment they cross partitions, retries, and parallel consumers, that guarantee evaporates. You can’t assume the order of receipt matches the order of occurrence.

Why ordering breaks

• Multiple partitions — a topic is split for throughput, and there’s no global order across partitions, only within one.
• Retries — a failed-then-retried message arrives after later messages that succeeded the first time.
• Parallel consumers — N workers processing concurrently finish in nondeterministic order.
• Multiple producers / clock skew — two sources interleave, and wall-clock timestamps disagree because clocks drift.

Order only where it actually matters

Global total order is expensive and usually unnecessary. What you almost always need is per-entity order: all events for account 42 in sequence, while unrelated accounts proceed in parallel. The lever is the partition key — partition by entity id, and a system like Kafka guarantees order within that partition. One key → one partition → one consumer → correct local order, with full parallelism across keys.

Tolerating reorder when you can’t prevent it

Where you can’t pin everything to one partition, make consumers robust to disorder:

• Versions / sequence numbers — stamp each event with a monotonically increasing version per entity. The consumer stores the last version applied and rejects anything older:

on event(e):
    if e.version <= stored_version[e.key]:
        drop  # stale or duplicate — a newer state already won
    else:
        apply(e); stored_version[e.key] = e.version

• Commutative / idempotent updates — design operations whose end state doesn’t depend on order (set-to-value with a version guard, add-to-set, max). Then reorder is harmless.
• A small reorder buffer — briefly hold events and sort by sequence before applying, accepting a little latency to restore order (the watermark lesson builds on this).

Where it shows up

Event sourcing and CQRS (replaying events to rebuild state), Kafka consumers, database replication (apply changes in commit order), CDC pipelines, and any multi-region system where writes converge from several sources.

The interview cue

When your design streams events, state the assumption: “Across partitions there’s no global order, so I’d partition by entity id to get per-key ordering, and make the consumer idempotent — apply only if the event’s version is newer than what I’ve recorded, so a late or duplicate event can’t overwrite fresher state.” Naming per-key ordering plus version-guarded idempotency shows you know receipt order is not occurrence order.