Fast, Frictionless, and Secure: Explore our 120+ Connectors Portfolio | Join Webinar!


The Log of All Logs: Raft-based Consensus Inside Kafka

« Kafka Summit APAC 2021

Kafka organizes data as immutable append-only logs at its core, and relied on external consensus services (a.k.a. Zookeeper) to manage the metadata --- such as topic-level configs, leader replicas and ISR information, received admin requests --- of these logs. In this talk, I will discuss a recent core initiative, that migrates the management of such metadata from external services into Kafka as its own special logs. More specifically, I will cover the following:

  1. Why we believe an internal consensus protocol provides Kafka more benefit than an external consensus service.
  2. Why we choose to build this internal "metadata log" based on the Raft protocol, instead of Kafka's current leader-follower replication mechanism.
  3. What are the key design decisions we made in its implementation, and how it is different from the standard Raft algorithm (KIP-595).
  4. How this Raft-based metadata log is leveraged by the new Quorum Controller (KIP-500).

Chinese Japanese Korean

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how