Confluent Cloud Q1 Launch: Build a Secure Shared Services Data Streaming Platform | Learn more

Should You Read Kafka as a Stream or in Batch? Should You Even Care?

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective?

In this talk we鈥檒l give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons.

To highlight the differences, we鈥檒l dive into a project we鈥檝e done, transitioning from reading Kafka in a stream to reading it in batch.

By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we鈥檝e achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution.

We鈥檒l explore the tradeoffs and give you the metrics and intuition you鈥檒l need to make such decisions yourself.

We鈥檒l cover:

  • Costs of processing in stream compared to batch
  • Scaling up for bursts and reprocessing
  • Making the tradeoff between wait times and costs
  • Recovering from outages
  • And much more...

Chinese Japanese Korean

Presentadores

Ido Nadler
Opher Dubrovsky