Jun RaoCo-founder of Confluent and Co-creator of Kafka
The Apache Kafka community just announced the 0.8.2.1 release. This is a a bug fix release and fixes 4 critical issues reported in the 0.8.2.0 release (the full list of features can be found here).
The first issue is that in certain cases, a broker can use significantly more CPU compared with the 0.8.1 release. This happens when (1) replication is used, (2) the number of partitions per broker is high (say a few thousand) and (3) the cluster is small (say less than 5 brokers).
The second issue is that when starting a broker with a conflicting broker id, the process of shutting down this broker accidentally removes critical broker registration in ZooKeeper. This may cause some of the topic partitions to be unavailable unexpectedly later on.
The last two issues are in the new Java producer. We fixed an issue that a message may not be delivered since the producer doesn’t correctly pick an available partition. This tends to happen when the replication factor is only one and one or more brokers fail. Another issue that we fixed is that on an empty Kafka cluster, the CPU usage of the new producer is high after initialization.
Since those issues are relatively critical, we recommend everyone upgrade to the 0.8.2.1 release. Note that all those changes are already included in the Kafka jar shipped with Confluent Platform 1.0.
Jun Rao is the co-founder of Confluent, a company that provides a stream data platform on top of Apache Kafka. Before Confluent, Jun Rao was a senior staff engineer at LinkedIn where he led the development of Kafka. Before LinkedIn, Jun Rao was a researcher at IBM's Almaden research data center, where he conducted research on database and distributed systems. Jun Rao is the PMC chair of Apache Kafka and a committer of Apache Cassandra.
The call for papers for Kafka Summit London 2023 has opened, and we’re looking to hear about your experiences using and working with Kafka. Every great technical talk starts with an experience. If you’re stuck looking for ideas on what to talk about, write what you know...
Rebalancing comes into play in Apache Kafka® when consumers join or leave a consumer group. In either case, there is a different number of consumers over which to distribute the partitions from the topic(s), and, so, they must be redistributed and rebalanced....