[Webinar] 4 Tips for Cutting Your Kafka Costs Up to 60%| Register Now

Presentation

Portable Streaming Pipelines with Apache Beam

« Kafka Summit New York City 2017

Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. By cleanly separating the user’s processing logic from details of the underlying execution engine, the same pipelines will run on any Apache Beam runtime environment, whether it’s on-premise or in the cloud, on open source frameworks like Apache Spark or Apache Flink, or on managed services like Google Cloud Dataflow.

In this talk, I will:

  • Briefly introduce the capabilities of the Beam model for data processing and integration with IO connectors like Apache Kafka.
  • Discuss the benefits Beam provides regarding portability and ease-of-use.
  • Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise).
  • Give a glimpse at some of the challenges Beam aims to address in the future.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how