[Webinar] Mastering Kafka Security Across Hybrid Environments → Register Now


A Kafkaesque Raft Protocol

« Kafka Summit Europe 2021

KIP-500 set the vision for Zookeeper-free Kafka. However, even without Zookeeper, the need for consensus never went away. In this talk, we will discuss one of the core community’s initiatives, a native Raft-like protocol used to ensure different brokers can agree on critical pieces of metadata such as which replicas are available for writing (i.e. partition leaders).

Specifically, we will cover the following topics:

  • Why did we abandon the external consensus and what benefits internal consensus provides.
  • How this protocol is different from standard Raft, and the critical design trade-offs we made in its implementation.
  • How the new Quorum Controller serves as the “Kafka control plane” and how it gets integrated with the Raft protocol
  • What next steps we envisage for Kafka’s replication protocol for metadata and beyond.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how