Kora Engine, Data Quality Rules und mehr in unserem Q2 2023 Launch | FĂŒr die Demo registrieren

Deep Dive Into Kafka Tiered Storage

« Current 2022

KIP-405 introduced tiered storage in Apache Kafka. The proposed design introduces the separation of compute and storage which benefits the brokers to largely focus on serving producer or consume requests and not manage the storage beyond local disks. But the important caveat here is that it should still maintain the same consistency semantics and lineage of data as in the local storage. This talk dives into the internals of tiered storage in how we achieve those semantics covering scenarios like new brokers bootstrapped, or brokers having hard failures, or other out-of-sync brokers becoming leaders etc.

We will also talk about how topic deletion lifecycle management is done without leaking any segments in tiered storage based on the retention policies or while deleting a topic or a partition.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how