Adopt Kafka with Confidence: Load and Stress Testing Kafka with Blizzard

Uber heavily relies on Kafka for its real-time data streaming and event-driven architecture. Kafka powers critical components of the platform such as ride dispatching, driver tracking, trip data processing, and real-time analytics. With the need for efficient benchmarking tools to optimize performance, adopt new version and capacity planning, Here at Uber we have developed a tool called Blizzard. Blizzard is a service that aims to be an end-to-end testing tool for stress testing Kafka clusters. It allows users to configure various parameters (like topic configuration, producer and consumer configurations) , simulate different production workloads which can be updated during testing, and fine-tune settings on the fly.

Blizzard leverages the uForwarder (https://github.com/uber/uForwarder) which is planned to be open-sourced from uber which provides a common framework for implementing low latency, high performance, high scale data transfer to and from Kafka. It follows a controller-worker model, where the controller handles admin actions like creation of topics and jobs and job management while the worker performs the actual production and consumption tasks. Blizzard offers features like cluster warm-up, dynamic scaling, load reproducibility, and the ability to mimic the current production workload. It also has pluggable implementation of producer and consumer which allows users to implement and choose between them.

We have used the tool internally to certify multiple SKUs, benchmark new builds and benchmark Uber's Kafka System in general.