Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more

Presentation

Optimizing for Low Latency and High Throughput

« Current 2022

A Sticky Partitioner was introduced in Kafka 2.4 to improve record batching, but it caused performance problems by directing more data to slower brokers. To fix this problem, a new Sticky Partitioner was developed to direct data to faster brokers to provide better latency and throughput for unkeyed messages, as described here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner.

In this talk I'll cover a simple, but effective algorithm for auto-tuning effective batch size for low latency and high throughput, adaptive partitioning logic to direct more data to faster brokers, and go through benchmark results that illustrate effectiveness of the new Sticky Partitioner.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how