[Webinar] Harnessing the Power of Data Streaming Platforms | Register Now


Handing Failure With Grace in Kafka Streams

¬ę Current 2022

Kafka Streams has recently expanded its options for handling thread death. Historically, upon reaching a fatal exception in a Streams Task, each thread would shut down, causing a rebalance. A different thread would then encounter the error as it picked up the task. This cascading thread failure could take a while. The performance suffers during the entire process due to constant rebalances and a non-optimal amount of threads. Eventually, there will be no threads alive, causing processing to halt entirely. Only then would the state change, alerting users to the issue.
In this talk, we will cover the changes to the threading model that made more dynamic error handling possible. We will also introduce the Streams handler, which unlocked options to react immediately in cases that would previously cause cascading thread death. Further improvements included modifying the state machine to clarify the meaning of the ERROR state. The inclusions of Kips 671, 696, and 663 allowed for much more flexibility in exceptional cases. After this talk, the audience can use the new handler to react to exceptional cases in Kafka Streams. They will also understand the updates to the threading model and the changes in the state machine.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how