The Dataflow Model Revisited

Opening excerpt

Eleven years ago, the Dataflow Model paper argued that unbounded, out-of-order data was the new normal, and that we must stop waiting for data to ever become complete. It proposed a unified model (windowing, triggers, watermarks, and retractions) for freely trading off correctness, latency, and cost across batch and streaming engines. On the occasion of its VLDB Test of Time award, we grade our own work—a paper about streaming analytics, in truth if not in name—on what aged well, what aged badly, and what we missed. We find the paper’s core foundations largely sound: the primacy of event time, the futility of waiting for completeness, and the insistence on strong consistency aged well.

(Excerpted from the original; full article via the source link below.)

This story hit the Hacker News front page today (73 points / 12 comments, via vldb.org). Our Tech Radar aggregates daily signals on AI engineering, backend architecture and DevOps — browse the related services and further reading below, or get in touch with our team.