Migrate from Kafka services to Confluent | Download Step-by-Step Guide

Presentation

Beyond Tiered Storage: Serverless Kafka with No Local Disks

« Kafka Summit London 2024

Separation of compute and storage has become the de-facto standard in the data industry for batch processing.

The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.

In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.

Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.

This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how