Register for Apache Kafka®, Confluent, and the Data Mesh

Kafka’s New Control Plane: The Quorum Controller

Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. In 2019, we outlined a plan to break this dependency and bring metadata management into Kafka itself through a dynamic service that runs inside the Kafka Cluster. We call this the Quorum Controller.

In this talk, we’ll look at how the Quorum Controller works and how it integrates with other parts of the next-generation Kafka architecture, such as the Raft quorum and snapshotting mechanism. We’ll also explain how the Quorum Controller will simplify operations, improve security, and enhance scalability and performance.

Finally, we’ll look at some of the practicalities, such as how to monitor and run the Quorum Controller yourself. We’ll talk about some of the performance gains we’ve seen, and our plans for the future.


Colin McCabe

Colin McCabe currently works for Confluent to improve the performance, scalability, and general awesomeness of Apache Kafka. Previously, he worked on the Apache™ Hadoop® Distributed Filesystem. Colin studied computer science and computer engineering at Carnegie Mellon University, and he enjoys homebrewing and scuba diving.