Improving Streams Scalability with Transactional StateStores (KIP-892)

Kafka Streams provides a horizontally scalable framework for exactly-once stream processing. But the more data a Kafka Streams app stores, the longer the state restore times become, which can grow into massive production incidents.

This talk will look at a key problem with the scalability of Kafka Streams, how it’s being resolved with KIP-892, Transactional StateStores, and future improvements that can be made to further improve the situation.