[Ebook] The Builder's Guide to Streaming Data Mesh | Read Now

Presentation

Kafka’s New Control Plane: The Quorum Controller

« Kafka Summit Europe 2021

Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. In 2019, we outlined a plan to break this dependency and bring metadata management into Kafka itself through a dynamic service that runs inside the Kafka Cluster. We call this the Quorum Controller.

In this talk, we’ll look at how the Quorum Controller works and how it integrates with other parts of the next-generation Kafka architecture, such as the Raft quorum and snapshotting mechanism. We’ll also explain how the Quorum Controller will simplify operations, improve security, and enhance scalability and performance.

Finally, we’ll look at some of the practicalities, such as how to monitor and run the Quorum Controller yourself. We’ll talk about some of the performance gains we’ve seen, and our plans for the future.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how