[Ebook] The Builder's Guide to Streaming Data Mesh | Read Now
Kafka Streams makes it very easy to run a distributed stream processing system: you just start multiple processes with the same ""application id"", point them at the same Kafka cluster, and they will form a distributed system on their own. Making a distributed system so easy to start up has an Achilles' heel: there's no natural learning curve that forces you to learn how to operate the system in order to stand it up.
Of all the pitfalls this situation creates, by far the most complex and user confusing is the topic of rebalancing and task assignment. Kafka Streams partition and task assignment is especially complex because of advanced features like ""smooth scale-out"" that uses warm-up tasks and so-called probing rebalances. This talk aims to completely demystify the system from an operational perspective. You will learn what is really happening across Kafka and Kafka Streams, how to interpret the logs and metrics, and how to adjust the configs to achieve your desired outcomes.