Introducing Streamhouse: the open data architecture for AI | Learn More
In most distributed systems, the clients are distributed along with the servers. Traditionally, benchmarking an Apache Kafka cluster has involved running the perf-test and e2e-latency CLI tools from a single machine. This approach is only valid when there is a single client instance using the Kafka cluster.
A typical benchmarking approach does not represent the real world traffic and is synthetic.
When the clients are distributed, concurrency is the devil, which is not accounted for by most benchmarking tools.
Traditional perf test tools also lack the ability to interpret, record and view the metrics for a period of time.
The Kafka Benchmark Operator is an open source Kubernetes operator to run the performance tests in a truly distributed fashion using the Kafka perf tools. The operator includes a Prometheus based scaffold to record the performance metrics, thereby supporting Prometheus Remote Write to systems such as New Relic, Elastic, Influx, etc.
In this session, we will discuss the essentials for an accurate and developer friendly benchmarking tool for Kafka and how the Kafka Benchmark Operator solves for this. We will take a look at other benchmarking tools for Kafka and discuss the merits and demerits of these tools. The session includes a live demo where we benchmark the cluster using the operator and demonstrate using the operator for various conditions. The attendees will be able to understand how to approach benchmarking an Apache Kafka cluster in a distributed manner and utilize the operator for fine tuning Apache Kafka clusters.