From 🐛 to 🦋: Data Pipelines Evolution from Batch to Streaming

Despite data streaming being in most companies’ agenda, transitioning out of consolidated batch systems is not as simple as flipping a switch: new technologies, processes and coding frameworks need to be assessed and then adopted which can make the evolution a long and painful process. But, what if we could keep the same framework? This session explores how Apache Flink can narrow the gap between batch and streaming by keeping the same data pipelines definition while the underlying technology evolves.

We’ll start the journey with a typical batch system, based on a relational database, and then showcase how to evolve it to streaming using Apache Flink and Apache Kafka with minimal changes on the data pipeline definition. We’ll cover query based connectors, mimicking the batch behavior, and then move to more advanced change data capture solutions with Debezium. Finally we’ll touch on critical topics like data validation and late arrival of events and expose strategies on how to minimize related risks.

If you’re thinking about migrating from batch to streaming, but are afraid of any disruption the process may cause in your organization, this session is for you!