Watermarks and late-arriving data

When is a window finished?

You want “page views per minute.” Easy in a batch — group by minute, count. In a stream it’s subtle: events for 10:00–10:01 don’t all arrive by 10:01. A phone was offline, a partition lagged, a retry took a detour, so a 10:00 event shows up at 10:03. If you emit the minute’s count the instant the clock ticks 10:01, you undercount. If you wait forever for stragglers, you never emit. You need a principled answer to “have I seen enough to close this window?” — that’s a watermark.

Event time vs processing time

Two clocks are in play. Event time is when the thing happened (stamped at the source); processing time is when your pipeline saw it. They diverge by the network/queue/retry delay. Aggregations almost always want event-time windows (“views in the 10:00 minute,” not “views my server received in its 10:00 minute”), because event time is stable and replayable while processing time depends on infrastructure hiccups.

The watermark

A watermark is a moving assertion: “I believe I’ve now seen all events with event time ≤ T.” When the watermark passes the end of a window, the window is declared complete and its result is emitted. The watermark is a heuristic, usually max event time seen so far − slack, where slack is how much lateness you expect:

events (event-time): 10:00:05  10:00:20  10:00:15  10:01:10  ...
                                              ^ watermark = 10:00:55 (max seen 10:01:10 − 15s slack)
window [10:00,10:01) fires when watermark >= 10:01:00

More slack → fewer events declared late, but every result is delayed. Less slack → fast results, more stragglers missed. The watermark is the completeness-vs-latency dial.

Late data — after the window already fired

Some events arrive past the watermark. Three options:

• Drop them — simplest; acceptable when a tiny tail doesn’t matter.
• Allowed lateness — keep the window’s state for an extra grace period and re-emit an updated result when a late event lands (downstream must handle restatements).
• Side output / dead-letter — route late events to a separate stream for inspection or batch reconciliation, so they’re not silently lost.

Where it shows up

Stream processors make this explicit: Flink and Beam/Dataflow have first-class watermarks, windows, triggers, and allowed-lateness; Spark Structured Streaming has withWatermark. Anything doing event-time windowed aggregation — real-time analytics, billing meters, fraud counters — lives or dies on getting the watermark right.

The interview cue

When a design aggregates a stream over time, separate the clocks and name the mechanism: “I’d window by event time, not processing time, and use a watermark — max seen − a few seconds — to decide when a window is complete. Late events past the watermark I’d handle with allowed-lateness restatements or a side output, and I’d tune the slack to trade completeness against latency.” Knowing that windowing needs a completeness signal, not just a clock, is the distinguishing detail.