Beyond Tiered Storage: Serverless Kafka with No Local Disks

Separation of compute and storage has become the de-facto standard in the data industry for batch processing.

The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.

In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.

Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.

This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero.