Kafka Streams has recently expanded its options for handling thread death. Historically, upon reaching a fatal exception in a Streams Task, each thread would shut down, causing a rebalance. A different thread would then encounter the error as it picked up the task. This cascading thread failure could take a while. The performance suffers during the entire process due to constant rebalances and a non-optimal amount of threads. Eventually, there will be no threads alive, causing processing to halt entirely. Only then would the state change, alerting users to the issue.
In this talk, we will cover the changes to the threading model that made more dynamic error handling possible. We will also introduce the Streams handler, which unlocked options to react immediately in cases that would previously cause cascading thread death. Further improvements included modifying the state machine to clarify the meaning of the ERROR state. The inclusions of Kips 671, 696, and 663 allowed for much more flexibility in exceptional cases. After this talk, the audience can use the new handler to react to exceptional cases in Kafka Streams. They will also understand the updates to the threading model and the changes in the state machine.