Découvrez de nombreuses nouveautés à l'occasion de notre lancement du 2e trimestre 2023 : moteur Kora Engine, règles de qualité des données, et bien plus encore | S'inscrire pour une démo
A Sticky Partitioner was introduced in Kafka 2.4 to improve record batching, but it caused performance problems by directing more data to slower brokers. To fix this problem, a new Sticky Partitioner was developed to direct data to faster brokers to provide better latency and throughput for unkeyed messages, as described here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner.
In this talk I'll cover a simple, but effective algorithm for auto-tuning effective batch size for low latency and high throughput, adaptive partitioning logic to direct more data to faster brokers, and go through benchmark results that illustrate effectiveness of the new Sticky Partitioner.