Implementing Exactly-once Delivery and Escaping Kafka Rebalance Storms

Even though stream processing has come a long way in the last few years, ensuring exactly-once delivery remains a difficult problem to solve.

This becomes an even bigger challenge when your consumers are distributed applications, and their Kubernetes pods can be scaled-out, scaled-in or simply restarted at any given moment, causing Apache Kafka to go into a “rebalance storm”.

In this talk, we’ll walk you through how we implemented exactly-once delivery with Kafka by managing Kafka transactions the right way, and how we escaped endless rebalance storms when running hundreds of consumers on the same Kafka topic.

We will discuss the issues we faced building Akamai’s data ingestion infrastructure on Azure, processing malicious traffic at internet scale.

The session covers:

• Kafka delivery semantics
• Kafka transactional API
• Kafka “anti-rebalance” tips and tricks