In most distributed systems, the clients are distributed along with the servers. Traditionally, benchmarking an Apache Kafka cluster has involved running the perf-test and e2e-latency CLI tools from a single machine. This approach is only valid when there is a single client instance using the Kafka cluster.
-
A typical benchmarking approach does not represent the real world traffic and is synthetic.
-
When the clients are distributed, concurrency is the devil, which is not accounted for by most benchmarking tools.
-
Traditional perf test tools also lack the ability to interpret, record and view the metrics for a period of time.
The Kafka Benchmark Operator is an open source Kubernetes operator to run the performance tests in a truly distributed fashion using the Kafka perf tools. The operator includes a Prometheus based scaffold to record the performance metrics, thereby supporting Prometheus Remote Write to systems such as New Relic, Elastic, Influx, etc.
In this session, we will discuss the essentials for an accurate and developer friendly benchmarking tool for Kafka and how the Kafka Benchmark Operator solves for this. We will take a look at other benchmarking tools for Kafka and discuss the merits and demerits of these tools. The session includes a live demo where we benchmark the cluster using the operator and demonstrate using the operator for various conditions. The attendees will be able to understand how to approach benchmarking an Apache Kafka cluster in a distributed manner and utilize the operator for fine tuning Apache Kafka clusters.
Presenter
Avinash Upadhyaya
PlatformatoryAvinash is a platform engineer and an experienced consultant helping business deploy, scale and architect data platforms using open source technologies such as Apache Kafka, Apache Flink, Kubernetes, etc. He is an avid technical evangelist advocating for open source technologies in the local communities through meetups and conferences. He helps ruin multiple local communities around data streaming, Apache Kafka, Apache Flink, Apache Pinot among others. Apart from consulting, Avinash is an open source contributor with contributions to various projects in the CNCF ecosystem.
Presenter
Ashwin Venkatesan
PlatformatoryI am a Data enthusiast working mainly in architecting Data platforms for scale and efficiency. I am passionate about real time data stream processing architectures. I come from a Data science background having previously worked in building Box Office prediction models for upcoming movies and scoring movie scripts using LLM models.
I currently consult various businesses in designing and building streaming data architectures with open source tools like Apache Kafka, Apache Flink etc. I have worked hands-on with major businesses across the globe to architect, deploy and optimise their production-grade Kafka clusters. I have experience building customised stream processing pipelines using Kafka Streams, KSQL and Apache Flink for a variety of use-cases.
I am actively involved in the Bengaluru Streams and Kafka community having given multiple talks on applications/use-cases around the Kafka ecosystem. My current interests are Real Time ML platforms, WASM based executions etc.