On behalf of the Apache Kafka® community, it is my pleasure to announce the release of Apache Kafka 3.1.0. The 3.1.0 release contains many improvements and new features. We’ll highlight some of the more prominent features in this blog post, but see the release notes for the full list of changes.
While KRaft is still not recommended for production (known gaps), we have fixed multiple bugs and we have continued to add missing features.
Tiered Storage work continues, with the goal of unlocking infinite scaling and faster rebalance times.
You can also watch the release video for a summary of what’s new in Apache Kafka 3.1.0:
Starting from Apache Kafka 3.1, the
FetchRequest supports topic IDs. Topic IDs provide a safer way to fetch data from topics without any chance of incorrectly interacting with stale topics with the same name. It also improves the efficiency of the fetch protocol because sending
Uuids on the wire is generally smaller than sending
Strings. This is a major step forward in the development of KIP-516.
KIP-773 enhances naming consistency for three new client metrics with millis and nanos. For example,
io-waittime-total is reintroduced as
io-wait-time-ns-total. The previously introduced metrics without
ns will be deprecated but available for backward compatibility.
KIP-768 provides a built-in and production-grade implementation of the interfaces defined in KIP-255 to allow Kafka to connect to an OpenID identity provider (e.g., Okta, Auth0, and Microsoft Azure) for authentication and token retrieval.
KIP-748 introduces two new metrics that are exposed by both the ZooKeeper and KRaft controller:
FencedBrokerCount. They respectively expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller.
The cooperative rebalancing protocol has been the default since Kafka 2.4, but we have continued to support the eager rebalancing protocol to provide an upgrade path from earlier client versions. This support will be dropped in a future release, so any users still on the eager protocol should prepare to finish upgrading their applications to the cooperative protocol in version 3.1. See KAFKA-13439 for more details.
KIP-783 guarantees that every exception thrown up to the uncaught exception handler, whether that be the new
StreamsUncaughtExceptionHandler or the old generic
UncaughtExceptionHandler, is wrapped as a
StreamsException. The KIP also introduces a new
TaskId field to the
StreamsException class, with a getter API to expose it. This field is set for any exception that originates from, or is tied to, a specific task.
Today, foreign-key (FK) joins in Kafka Streams only work if both tables being joined (the primary table and the foreign-key table) use the default partitioner.
This limitation is due to the subscription and response topics in the implementation being hardwired to use the default partitioner. If the foreign-key table is not co-partitioned with the subscription topic, then foreign-key lookups may be routed to a Streams instance that does not have state for the foreign-key table, resulting in missing join records. Similarly, if the primary table is not co-partitioned with the response topic, then subscription responses may be routed to an instance that does not contain the original (triggering) record, resulting in a failed hash comparison and a dropped join result.
KIP-775 introduces support for foreign-key joins on tables with custom partitioners, by extending the foreign-key join interface to allow custom partitioners to be passed in.
KIP-766 extends the semantics of the existing range interfaces in the
ReadOnlySessionStore and the
ReadOnlyWindowStore to support unbounded ranges. Specifically, the interfaces now support the use of
null values as a way to represent unbounded ranges.
KIP-763 extends the semantics of the existing
reverseRange interfaces in the
ReadOnlyKeyValueStore to support unbounded ranges. Specifically, the interfaces now support the use of null values as a way to represent unbounded ranges.
KIP-761 introduces a new metric called
blocked-time-total that measures the total time a Kafka Streams thread has spent blocked on Kafka since it was started. Users can sample this metric periodically and use the difference between samples to measure time blocked during an interval. This is very useful to debug Kafka Streams application performance as it gives the proportion of time the application was blocked on Kafka vs. processing records.
MirrorMaker2 (MM2) internal topic names (heartbeats, checkpoints, and offset syncs) are hardcoded in the source code, which makes it hard to run MM2 with any Kafka cluster that has rules around a topic’s naming convention and doesn’t allow auto-creation for topics. In this case, you will need to create these internal topics upfront manually and make sure they do follow the cluster rules and guidance for topic creation, so MM2 should have flexibility to let you override the name of internal topics to use the ones you create.
Apache Kafka 3.1 has a lot of great fixes and improvements in addition to the KIPs listed here. To learn more:
This was a huge community effort, so thank you to everyone who contributed to this release, including all our users and our 114 authors and reviewers:
A. Sophie Blee-Goldman, Alexander Iskuskov, Alexander Stohr, Almog Gavra, Andras Katona, Andrew Patterson, Andy Chambers, Andy Lapidas, Anna Sophie Blee-Goldman, Antony Stubbs, Arjun Satish, Bill Bejeck, Boyang Chen, Bruno Cadonna, CHUN-HAO TANG, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez, Ismael Juma, Israel Ekpo, Ivan Ponomarev, Jakub Scholz, James Galasyn, Jason Gustafson, Jeff Kim, Jim Galasyn, JoeCqupt, Joel Hamill, John Gray, John Roesler, Jongho Jeon, Jorge Esteban Quilcate Otoya, Jose Sancio, Josep Prat, José Armando García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal Chandraprakash, Kevin Zhang, Kirk True, Konstantine Karantasis, Kowshik Prakasam, Leah Thomas, Lee Dongjin, Lucas Bradstreet, Luke Chen, Manikumar Reddy, Matthew Wong, Matthias J. Sax, Michael Carter, Mickael Maison, Nigel Liang, Niket, Niket Goel, Oliver Hutchison, Omnia G H Ibrahim, Patrick Stuedi, Phil Hardwick, Prateek Agarwal, Rajini Sivaram, Randall Hauch, René Kerner, Richard Yu, Rohan, Ron Dagostino, Ryan Dielhenn, Sanjana Kaundinya, Satish Duggana, Sergio Peña, Sherzod Mamadaliev, Stanislav Vodetskyi, Ted Yu, Tom Bentley, Tomas Forsman, Tomer Wizman, Uwe Eisele, Victoria Xia, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Weisheng Yang, Xavier Léauté, Yanwen(Jason) Lin, Yi Ding, Zara Lim, andy0x01, dengziming, feyman2016, ik, ik.lim, jem, jiangyuan, kpatelatwork, leah, loboya~, lujiefsi, sebbASF, singingMan, vamossagar12, wenbingshen
This post was originally published by David Jacot on The Apache Software Foundation blog.
David Jacot is a staff engineer on the Kafka Team at Confluent, where he works on building the foundations of the next-generation, cloud-native event streaming platform. He is an Apache Kafka Committer and a member of the Apache Kafka Program Management Committee (PMC). In his spare time, he relaxes by playing with his daughters and reviewing PRs.