What’s New in Apache Kafka 3.4

The Apache Kafka® community is proud to announce the release of Apache Kafka 3.4. The 3.4 release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

One of the highlights of this release is the ability to migrate Kafka clusters from ZK to KRaft mode with no downtime. This is an early access feature, which is currently only suitable for testing in non production environments. It is another key step on the journey towards KRaft mode. See KIP-866 for more details.

Kafka Broker, Controller, Producer, Consumer and Admin Client

KIP-866: ZooKeeper to KRaft Migration (Early Access)

KIP-866 adds the capability to migrate a Kafka cluster from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. ZooKeeper to KRaft migrations are Early Access in 3.4 and should not be used in production yet, though we are looking for people to provide feedback on this feature. Notably, support for migrating ACLs and downgrading to ZooKeeper mode during or after the migration is not supported in 3.4.

KIP-792: Add generation field into consumer protocol

KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation that consumer was a part of. This information can be used by the partition assignor to determine whether the ownedPartitions claimed by a consumer are still owned and thus valid, and which consumer has the most recent claim to a given partition in case no one has a valid claim to it.

KIP-830: Allow disabling JMX Reporter

KIP-830 includes a new configuration setting that allows you to disable the JMX reporter for environments where it’s not being used. 

KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

KIP-840 provides a new config file option for configuring the MessageReader/Formatter of the console Consumer/Producer by introducing the new parameters --reader-config and --formatter-config.

KIP-854: Separate configuration for producer ID expiry

KIP-854 introduces changes to clean up producer IDs more efficiently, to avoid excess memory usage. It introduces a new timeout parameter, with a default of 1 day, that affects the expiry of producer IDs and updates the old parameter to only affect the expiry of transaction IDs.

KIP-876: Time based cluster metadata snapshots

KIP-876 adds a new property that defines the maximum amount of time that the server will wait to generate a snapshot; the default is 1 hour. This feature was only implemented for the KRaft Controllers in 3.4.

KIP-881: Rack-aware Partition Assignment for Kafka Consumers

KIP-881 is an extension of KIP-392, which makes it so that consumers can be rack-aware when it comes to partition assignments and consumer rebalancing. Note that only the protocol changes have been implemented in 3.4.0 but none of the out-of-the-box assignors utilize this yet, so the rack-aware assignment is only available to those who plug in a custom assignor at this point.

Kafka Streams

KIP-770: Replace cache.max.bytes.buffering with cache.max.bytes

KIP-770 deprecates the existing cache.max.bytes.buffering config and introduces a new cache.max.bytes config to replace it. The semantics and default value of the cache size config is unchanged. This KIP also adds a new cache.size metric at the DEBUG level for users to monitor the actual size of the Kafka Streams cache.

KIP-837: Allow MultiCasting a Result Record

KIP-837 allows users to multicast result records to every partition of downstream sink topics and adds functionality for users to choose to drop result records without sending.

KIP-865: Support --bootstrap-server in kafka-streams-application-reset

KIP-865 updates the Kafka Streams application reset tool’s server parameter name to conform to the other Kafka tooling by deprecating the --bootstrap-servers parameter and introducing a new --bootstrap-server one in its place.

Kafka Connect

KIP-787: MirrorMaker2 custom Kafka Admin interface for managing Kafka resources

KIP-787 allows users to run MirrorMaker2 with custom implementations for the Kafka resource manager and makes it easier to integrate with your ecosystem.  This allows users to load a custom implementation of the Kafka Admin interface through the use of a new ForwardingAdmin class delegate.  A custom resource implementation solves the issues of providing MM2 with the correct authorization and authentication in organizations that use federated or resource management systems. 

Summary

In addition to all of the KIPs listed above, Apache Kafka 3.4 includes a number of fixes and improvements. To learn more:

• See the release notes for 3.4.0 for the full list of changes

• Watch the video

• Listen to the podcast

• Download Apache Kafka 3.4 and get started

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 117 authors:

 A. Sophie Blee-Goldman, Ahmed Sobeh, Akhilesh C, Akhilesh Chaganti, Alan Sheinberg, aLeX, Alex Sorokoumov, Alexandre Garnier, Alyssa Huang, Andras Katona, Andrew Borley, Andrew Dean, andymg3, Artem Livshits, Ashmeet Lamba, Badai Aqrandista, Bill Bejeck, Bruno Cadonna, Calvin Liu, Chase Thomas, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Christopher L. Shannon, Colin P. McCabe, Colin Patrick McCabe, Dalibor Plavcic, Dan Stelljes, Daniel Fonai, David Arthur, David Jacot, David Karlsson, David Mao, dengziming, Derek Troy-West, Divij Vaidya, Edoardo Comar, Elkhan Eminov, Eugene Tolbakov, Federico Valeri, Francesco Nigro, FUNKYE, Greg Harris, Guozhang Wang, Hao Li, Himani Arora, Huilin Shi, Igor Soarez, Ismael Juma, James Hughes, Janik Dotzel, Jason Gustafson, Jeff Kim, Jim Galasyn, JK-Wang, Joel Hamill, John Roesler, Jonathan Albrecht, Jordan Bull, Jorge Esteban Quilcate Otoya, José Armando García Sancio, Justine Olshan, K8sCat, Kirk True, Kvicii, Levani Kokhreidze, Liam Clarke-Hutchinson, LinShunKang, liuzc9, liuzhuang2017, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Matthew de Detrich, Matthew Stidham, Matthias J. Sax, Mickael Maison, Nandini Anagondi, Nick Telford, nicolasguyomar, Niket, Niket Goel, Nikolay, Okada Haruki, Oliver Eikemeier, Omnia G H Ibrahim, Orsák Maroš, Patrik Marton, Peter Nied, Philip Nee, Philipp Trulson, Pratim SC, Proven Provenzano, Purshotam Chauhan, Rajini Sivaram, Ramesh, Rens Groothuijsen, RivenSun, Rohan, Ron Dagostino, runom, Sanjana Kaundinya, Satish Duggana, Shawn, Shay Lin, Shenglong Zhang, srishti-saraswat, Stanislav Vodetskyi, Sushant Mahajan, Tom Bentley, vamossagar12, venkatteki, Vicky Papavasileiou, Walker Carlson, Yash Mayya, zou shengfu, 行路难行路