Confluent Platform 7.0 と Cluster Linking でクラウドへのリアルタイムブリッジを構築 | ブログを読む

Big Data Kappa

In a world where there is an ever growing shift towards event driven streaming data, Kafka is firmly embedded in the epicenter of any Data Platform’s central nervous system. In an attempt to aide in the shift of analytics towards true event time, we have implemented a pure Kappa architecture - effectively turning the database inside out. Through extending the concept of a truly idempotent stream of events, Kafka has been elevated to the source of truth. We have eliminated extra network trips for joins as well as querying state which has significantly improved processing performance while also reducing processing latency.

Tune in to discuss challenges, tips and lessons learned while implementing a pure Kappa Architecture. I will address hurdles such as scaling, warm standbys, schema evolution, and batch replay strategies - highlighting issues prevalent with any streaming Kappa based architecture. Streaming big data in and of itself comes with its own set of challenges - such as serialization formats, encryption, and strategies to efficiently utilize message headers. I invite each and every one of you to embark on a journey discussing a means to an end - resulting in processing billions of records each day.

プレゼンター

Mark Senerth

Mark Senerth has been solving problems for The Walt Disney Company for 15 years. He is well versed in cloud architecture, big data, distributed computing, and high throughput / low latency systems. Mark has the unique ability to see things from both a Macro and Micro perspective simultaneously; from the highest level architecture down the lowest level minute implementation detail. He has worked on a set of diverse datasets over the years that have been stored in an array of storage mediums - each having their own corresponding problem space. Currently Mark is focused on the Architecture and Implementation of the DMED (Disney Media & Entertainment Distribution) Data Tech Unified Data Platform. The platform will enable the business to move faster and achieve better insights in a secure, governed ecosystem.