Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing.
In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.
プレゼンター
Yaroslav Tkachenko
Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, data-intensive applications, modern cloud infrastructure, and DevOps practices.
Currently, Yaroslav is a Principal Software Engineer at Goldsky, focused on building a read layer for the blockchain data leveraging the power of stream-processing.
Before that, Yaroslav was a Staff Data Engineer at Shopify, working on building and supporting libraries, tools and services for Shopify's stream-processing use-cases. Previously, he was a Senior Software Engineer and later Software Architect at Activision, where he redesigned and rebuilt the data pipeline for Activision games like the Call of Duty franchise. Before joining Activision, Yaroslav held various leadership roles in multiple startups