White Paper

Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka®

Get the White Paper

An increasingly important system requirement for distributed stream processing applications is to provide strong correctness guarantees under unexpected failures and out-of-order data so that its results can be authoritative (not needing complementary batch results). Although existing systems have put a lot of effort into addressing some specific issues, such as consistency and completeness, how to enable users to make flexible and transparent trade-off decisions among correctness, performance, and cost still remains a practical challenge. Specifically, similar mechanisms are usually applied to tackle both consistency and completeness, which can result in unnecessary performance penalties.

This paper presents Apache Kafka’s core design for stream processing, which relies on its persistent log architecture as the storage and inter-processor communication layers to achieve correctness guarantees. Kafka Streams, a scalable stream processing client library in Apache Kafka, defines the processing logic as read-process-write cycles in which all processing state updates and result outputs are captured as log appends. Idempotent and transactional write protocols are utilized to guarantee exactly-once semantics. Furthermore, revision-based speculative processing is employed to emit results as soon as possible while handling out-of-order data. This paper also demonstrates how Kafka Streams behaves in practice with large-scale deployments and performance insights exhibiting its flexible and low-overhead trade-offs.

This paper was authored by Guozhang Wang, Lei Chen, Ayusman Dikshit, Jason Gustafson, Boyang Chen, Matthias J. Sax, John Roesler, Sophie Blee-Goldman, Bruno Cadonna, Apurva Mehta, Varun Madan, and Jun Rao.


cc demo

Confluent Cloud Demo

Join us for a live demo of Confluent Cloud, the industry’s only fully managed, cloud-native event streaming platform powered by Apache Kafka
kafka microservices

Kafka Microservices

In this online talk series, learn key concepts, use cases and best practices to harness the power of real-time streams for microservices architectures
Image-Event-Driven Microservices-01

e-book: Microservices Customer Stories

See how five organizations across a wide range of industries leveraged Confluent to build a new class of event-driven microservices