Jun RaoCo-founder of Confluent and Co-creator of Kafka
The Apache Kafka community just announced the 0.8.2.1 release. This is a a bug fix release and fixes 4 critical issues reported in the 0.8.2.0 release (the full list of features can be found here).
The first issue is that in certain cases, a broker can use significantly more CPU compared with the 0.8.1 release. This happens when (1) replication is used, (2) the number of partitions per broker is high (say a few thousand) and (3) the cluster is small (say less than 5 brokers).
The second issue is that when starting a broker with a conflicting broker id, the process of shutting down this broker accidentally removes critical broker registration in ZooKeeper. This may cause some of the topic partitions to be unavailable unexpectedly later on.
The last two issues are in the new Java producer. We fixed an issue that a message may not be delivered since the producer doesn’t correctly pick an available partition. This tends to happen when the replication factor is only one and one or more brokers fail. Another issue that we fixed is that on an empty Kafka cluster, the CPU usage of the new producer is high after initialization.
Since those issues are relatively critical, we recommend everyone upgrade to the 0.8.2.1 release. Note that all those changes are already included in the Kafka jar shipped with Confluent Platform 1.0.
Jun Rao is the co-founder of Confluent, a company that provides a stream data platform on top of Apache Kafka. Before Confluent, Jun Rao was a senior staff engineer at LinkedIn where he led the development of Kafka. Before LinkedIn, Jun Rao was a researcher at IBM's Almaden research data center, where he conducted research on database and distributed systems. Jun Rao is the PMC chair of Apache Kafka and a committer of Apache Cassandra.
As companies increase their use of real-time data, we have seen the proliferation of Kafka clusters within many enterprises. Often, siloed application and infrastructure teams set up and manage new clusters to solve new use cases as they arise. In many large, complex enterprises, this organic growth