It has been about 7 years since we introduced the so-called new consumer which does group membership and rebalancing through Apache Kafka. Although it was a huge improvement over the old Zookeeper-based consumer, it has proven to be a major pain point from an operation perspective.
This talk will unveil the next generation of the consumer rebalance protocol for Apache Kafka (KIP-848) that addresses the shortcomings of the current protocol. This new protocol has three major benefits:
(1) The stop-the-world rebalances are truly gone. We have designed an incremental rebalance protocol which drastically improves the stability of consumer groups. One misbehaving consumer can no longer take down the entire group.
(2) Unlike the current protocol which hardly scales to thousands of consumers within a single group, the new one is designed from the ground up to be scalable.
(3) The new rebalance protocol reduces the complexity of the consumers by bringing it back to the server side. This will simplify the implementation of the consumers (in all languages) and give operators all the visibility that they need to debug issues.
We will go through the evolution of the current rebalance protocol, discuss its shortcomings, and present the new rebalance protocol in detail.