Even though almost all production data is continuously processed, streaming is still not yet the default way these systems are built. In spite of significant advances, some of the most popular solutions and architectures for data and application integration, ETL, database replication and synchronization, backup, and process triggering remain batch processes at their core. Streaming is generally regarded as “nice to have,” too complex, or simply not a requirement even though it is arguably the obvious way to continuously process data. All things being equal, few would object to fresher data and more timely processing, so where is the disconnect?
This talk explores the current state of streaming, the most common objections and the reasons behind them, the massive technical and financial drag this has created, and what needs to change before streaming becomes the default way we process continuous data. Attendees will leave with a better understanding of common challenges, their impact, and what the future of streaming might look like along with some tips on how you can improve your data platform today.