Kora Engine, Data Quality Rules und mehr in unserem Q2 2023 Launch | FĂŒr die Demo registrieren

The Next Generation of the Consumer Rebalance Protocol

« Current 2022

It has been about 7 years since we introduced the so-called new consumer which does group membership and rebalancing through Apache Kafka. Although it was a huge improvement over the old Zookeeper-based consumer, it has proven to be a major pain point from an operation perspective.

This talk will unveil the next generation of the consumer rebalance protocol for Apache Kafka (KIP-848) that addresses the shortcomings of the current protocol. This new protocol has three major benefits:

(1) The stop-the-world rebalances are truly gone. We have designed an incremental rebalance protocol which drastically improves the stability of consumer groups. One misbehaving consumer can no longer take down the entire group.

(2) Unlike the current protocol which hardly scales to thousands of consumers within a single group, the new one is designed from the ground up to be scalable.

(3) The new rebalance protocol reduces the complexity of the consumers by bringing it back to the server side. This will simplify the implementation of the consumers (in all languages) and give operators all the visibility that they need to debug issues.

We will go through the evolution of the current rebalance protocol, discuss its shortcomings, and present the new rebalance protocol in detail.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how