Show Me How: Build Streaming Data Pipelines for Real-Time Data Warehousing | Register Today
We are proud to announce the release of Apache Kafka® 3.4 on behalf of the Apache Kafka community. The 3.4 release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.
One of the highlights of this release is the ability to migrate Kafka clusters from ZK to KRaft mode with no downtime. This is an early access feature, which is currently only suitable for testing in non production environments. It is another key step on the journey towards KRaft mode. See KIP-866 for more details.
KIP-866 adds the capability to migrate a Kafka cluster from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. ZooKeeper to KRaft migrations are Early Access in 3.4 and should not be used in production yet, though we are looking for people to provide feedback on this feature. Notably, support for migrating ACLs and downgrading to ZooKeeper mode during or after the migration is not supported in 3.4.
generationfield into consumer protocol
KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation that consumer was a part of. This information can be used by the partition assignor to determine whether the
ownedPartitions claimed by a consumer are still owned and thus valid, and which consumer has the most recent claim to a given partition in case no one has a valid claim to it.
KIP-830 includes a new configuration setting that allows you to disable the JMX reporter for environments where it’s not being used.
KIP-840 provides a new config file option for configuring the MessageReader/Formatter of the console Consumer/Producer by introducing the new parameters
KIP-854 introduces changes to clean up producer IDs more efficiently, to avoid excess memory usage. It introduces a new timeout parameter, with a default of 1 day, that affects the expiry of producer IDs and updates the old parameter to only affect the expiry of transaction IDs.
KIP-876 adds a new property that defines the maximum amount of time that the server will wait to generate a snapshot; the default is 1 hour. This feature was only implemented for the KRaft Controllers in 3.4.
KIP-881 is an extension of KIP-392, which makes it so that consumers can be rack-aware when it comes to partition assignments and consumer rebalancing. Note that only the protocol changes have been implemented in 3.4.0 but none of the out-of-the-box assignors utilize this yet, so the rack-aware assignment is only available to those who plug in a custom assignor at this point.
KIP-770 deprecates the existing
cache.max.bytes.buffering config and introduces a new
cache.max.bytes config to replace it. The semantics and default value of the cache size config is unchanged. This KIP also adds a new
cache.size metric at the DEBUG level for users to monitor the actual size of the Kafka Streams cache.
KIP-837 allows users to multicast result records to every partition of downstream sink topics and adds functionality for users to choose to drop result records without sending.
KIP-865 updates the Kafka Streams application reset tool’s server parameter name to conform to the other Kafka tooling by deprecating the
s parameter and introducing a new
--bootstrap-server one in its place.
KIP-787 allows users to run MirrorMaker2 with custom implementations for the Kafka resource manager and makes it easier to integrate with your ecosystem. This allows users to load a custom implementation of the Kafka Admin interface through the use of a new ForwardingAdmin class delegate. A custom resource implementation solves the issues of providing MM2 with the correct authorization and authentication in organizations that use federated or resource management systems.
In addition to all of the KIPs listed above, Apache Kafka 3.4 includes a number of fixes and improvements. To learn more:
See the release notes for 3.4.0 for the full list of changes
Download Apache Kafka 3.4 and get started
This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 117 authors:
A. Sophie Blee-Goldman, Ahmed Sobeh, Akhilesh C, Akhilesh Chaganti, Alan Sheinberg, aLeX, Alex Sorokoumov, Alexandre Garnier, Alyssa Huang, Andras Katona, Andrew Borley, Andrew Dean, andymg3, Artem Livshits, Ashmeet Lamba, Badai Aqrandista, Bill Bejeck, Bruno Cadonna, Calvin Liu, Chase Thomas, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Christopher L. Shannon, Colin P. McCabe, Colin Patrick McCabe, Dalibor Plavcic, Dan Stelljes, Daniel Fonai, David Arthur, David Jacot, David Karlsson, David Mao, dengziming, Derek Troy-West, Divij Vaidya, Edoardo Comar, Elkhan Eminov, Eugene Tolbakov, Federico Valeri, Francesco Nigro, FUNKYE, Greg Harris, Guozhang Wang, Hao Li, Himani Arora, Huilin Shi, Igor Soarez, Ismael Juma, James Hughes, Janik Dotzel, Jason Gustafson, Jeff Kim, Jim Galasyn, JK-Wang, Joel Hamill, John Roesler, Jonathan Albrecht, Jordan Bull, Jorge Esteban Quilcate Otoya, José Armando García Sancio, Justine Olshan, K8sCat, Kirk True, Kvicii, Levani Kokhreidze, Liam Clarke-Hutchinson, LinShunKang, liuzc9, liuzhuang2017, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Matthew de Detrich, Matthew Stidham, Matthias J. Sax, Mickael Maison, Nandini Anagondi, Nick Telford, nicolasguyomar, Niket, Niket Goel, Nikolay, Okada Haruki, Oliver Eikemeier, Omnia G H Ibrahim, Orsák Maroš, Patrik Marton, Peter Nied, Philip Nee, Philipp Trulson, Pratim SC, Proven Provenzano, Purshotam Chauhan, Rajini Sivaram, Ramesh, Rens Groothuijsen, RivenSun, Rohan, Ron Dagostino, runom, Sanjana Kaundinya, Satish Duggana, Shawn, Shay Lin, Shenglong Zhang, srishti-saraswat, Stanislav Vodetskyi, Sushant Mahajan, Tom Bentley, vamossagar12, venkatteki, Vicky Papavasileiou, Walker Carlson, Yash Mayya, zou shengfu, 行路难行路
Why replace ZooKeeper with an internal log for Apache Kafka® metadata management? This post explores the rationale behind the replacement, examines why a quorum-based consensus protocol like Raft was utilized […]
When you encounter a problem with Apache Kafka®—for example, an exploding number of connections to your brokers or perhaps some wonky record batching—it’s easy to consider these issues as something to be solved in and of themselves...