Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics.
In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently.
We will discuss:
- Kafka's spin on standard two-phase commit protocols
- How transaction state is maintained and replicated
- How different failure scenarios are handled
- What future improvements will make exactly-once stream processing with Kafka even better