A Kafkaesque Raft Protocol

KIP-500 set the vision for Zookeeper-free Kafka. However, even without Zookeeper, the need for consensus never went away. In this talk, we will discuss one of the core community’s initiatives, a native Raft-like protocol used to ensure different brokers can agree on critical pieces of metadata such as which replicas are available for writing (i.e. partition leaders).

Specifically, we will cover the following topics:

• Why did we abandon the external consensus and what benefits internal consensus provides.
• How this protocol is different from standard Raft, and the critical design trade-offs we made in its implementation.
• How the new Quorum Controller serves as the “Kafka control plane” and how it gets integrated with the Raft protocol
• What next steps we envisage for Kafka’s replication protocol for metadata and beyond.