What’s New in Apache Kafka 3.1.0

On behalf of the Apache Kafka^® community, it is my pleasure to announce the release of Apache Kafka 3.1.0. The 3.1.0 release contains many improvements and new features. We’ll highlight some of the more prominent features in this blog post, but see the release notes for the full list of changes.

While KRaft is still not recommended for production (known gaps), we have fixed multiple bugs and we have continued to add missing features.

Tiered Storage work continues, with the goal of unlocking infinite scaling and faster rebalance times.

You can also watch the release video for a summary of what’s new in Apache Kafka 3.1.0:

Kafka broker, Producer, Consumer and AdminClient

KIP-516: Topic identifiers

Starting from Apache Kafka 3.1, the FetchRequest supports topic IDs. Topic IDs provide a safer way to fetch data from topics without any chance of incorrectly interacting with stale topics with the same name. It also improves the efficiency of the fetch protocol because sending Uuids on the wire is generally smaller than sending Strings. This is a major step forward in the development of KIP-516.

KIP-773: Differentiate consistently metric latency measured in millis and nanos

KIP-773 enhances naming consistency for three new client metrics with millis and nanos. For example, io-waittime-total is reintroduced as io-wait-time-ns-total. The previously introduced metrics without ns will be deprecated but available for backward compatibility.

KIP-768: Extend SASL/OAUTHBEARER with support for OIDC

KIP-768 provides a built-in and production-grade implementation of the interfaces defined in KIP-255 to allow Kafka to connect to an OpenID identity provider (e.g., Okta, Auth0, and Microsoft Azure) for authentication and token retrieval.

KIP-748: Add broker count metrics

KIP-748 introduces two new metrics that are exposed by both the ZooKeeper and KRaft controller: ActiveBrokerCount and FencedBrokerCount. They respectively expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller.

Kafka Streams

KAFKA-13439: The eager rebalance protocol is deprecated

The cooperative rebalancing protocol has been the default since Kafka 2.4, but we have continued to support the eager rebalancing protocol to provide an upgrade path from earlier client versions. This support will be dropped in a future release, so any users still on the eager protocol should prepare to finish upgrading their applications to the cooperative protocol in version 3.1. See KAFKA-13439 for more details.

KIP-783: Add TaskId field to StreamsException

KIP-783 guarantees that every exception thrown up to the uncaught exception handler, whether that be the new StreamsUncaughtExceptionHandler or the old generic UncaughtExceptionHandler, is wrapped as a StreamsException. The KIP also introduces a new TaskId field to the StreamsException class, with a getter API to expose it. This field is set for any exception that originates from, or is tied to, a specific task.

KIP-775: Custom partitioners in foreign-key joins

Today, foreign-key (FK) joins in Kafka Streams only work if both tables being joined (the primary table and the foreign-key table) use the default partitioner.

This limitation is due to the subscription and response topics in the implementation being hardwired to use the default partitioner. If the foreign-key table is not co-partitioned with the subscription topic, then foreign-key lookups may be routed to a Streams instance that does not have state for the foreign-key table, resulting in missing join records. Similarly, if the primary table is not co-partitioned with the response topic, then subscription responses may be routed to an instance that does not contain the original (triggering) record, resulting in a failed hash comparison and a dropped join result.

KIP-775 introduces support for foreign-key joins on tables with custom partitioners, by extending the foreign-key join interface to allow custom partitioners to be passed in.

KIP-766: fetch/findSessions queries with open endpoints for SessionStore/WindowStore

KIP-766 extends the semantics of the existing range interfaces in the ReadOnlySessionStore and the ReadOnlyWindowStore to support unbounded ranges. Specifically, the interfaces now support the use of null values as a way to represent unbounded ranges.

KIP-763: Range queries with open endpoints

KIP-763 extends the semantics of the existing range and reverseRange interfaces in the ReadOnlyKeyValueStore to support unbounded ranges. Specifically, the interfaces now support the use of null values as a way to represent unbounded ranges.

KIP-761: Add total blocked time metric to Streams

KIP-761 introduces a new metric called blocked-time-total that measures the total time a Kafka Streams thread has spent blocked on Kafka since it was started. Users can sample this metric periodically and use the difference between samples to measure time blocked during an interval. This is very useful to debug Kafka Streams application performance as it gives the proportion of time the application was blocked on Kafka vs. processing records.

MirrorMaker

KIP-690: Add additional configuration to control MirrorMaker2 internal topics naming convention

MirrorMaker2 (MM2) internal topic names (heartbeats, checkpoints, and offset syncs) are hardcoded in the source code, which makes it hard to run MM2 with any Kafka cluster that has rules around a topic’s naming convention and doesn’t allow auto-creation for topics. In this case, you will need to create these internal topics upfront manually and make sure they do follow the cluster rules and guidance for topic creation, so MM2 should have flexibility to let you override the name of internal topics to use the ones you create.

KIP-690 introduces new methods to ReplicationPolicy that define how MM2 internal topics are named based on some new configuration.

Summary

Apache Kafka 3.1 has a lot of great fixes and improvements in addition to the KIPs listed here. To learn more:

• See the release notes for the full list of changes
• Check out the video or the podcast to learn more
• Download Apache Kafka 3.1 to get started with the latest release

This was a huge community effort, so thank you to everyone who contributed to this release, including all our users and our 114 authors and reviewers:

A. Sophie Blee-Goldman, Alexander Iskuskov, Alexander Stohr, Almog Gavra, Andras Katona, Andrew Patterson, Andy Chambers, Andy Lapidas, Anna Sophie Blee-Goldman, Antony Stubbs, Arjun Satish, Bill Bejeck, Boyang Chen, Bruno Cadonna, CHUN-HAO TANG, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez, Ismael Juma, Israel Ekpo, Ivan Ponomarev, Jakub Scholz, James Galasyn, Jason Gustafson, Jeff Kim, Jim Galasyn, JoeCqupt, Joel Hamill, John Gray, John Roesler, Jongho Jeon, Jorge Esteban Quilcate Otoya, Jose Sancio, Josep Prat, José Armando García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal Chandraprakash, Kevin Zhang, Kirk True, Konstantine Karantasis, Kowshik Prakasam, Leah Thomas, Lee Dongjin, Lucas Bradstreet, Luke Chen, Manikumar Reddy, Matthew Wong, Matthias J. Sax, Michael Carter, Mickael Maison, Nigel Liang, Niket, Niket Goel, Oliver Hutchison, Omnia G H Ibrahim, Patrick Stuedi, Phil Hardwick, Prateek Agarwal, Rajini Sivaram, Randall Hauch, René Kerner, Richard Yu, Rohan, Ron Dagostino, Ryan Dielhenn, Sanjana Kaundinya, Satish Duggana, Sergio Peña, Sherzod Mamadaliev, Stanislav Vodetskyi, Ted Yu, Tom Bentley, Tomas Forsman, Tomer Wizman, Uwe Eisele, Victoria Xia, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Weisheng Yang, Xavier Léauté, Yanwen(Jason) Lin, Yi Ding, Zara Lim, andy0x01, dengziming, feyman2016, ik, ik.lim, jem, jiangyuan, kpatelatwork, leah, loboya~, lujiefsi, sebbASF, singingMan, vamossagar12, wenbingshen

This post was originally published by David Jacot on The Apache Software Foundation blog.