Introducing Streamhouse: the open data architecture for AI | Learn More
In today's rapidly evolving digital landscape, the shift towards event-driven architecture is becoming increasingly prominent. This architectural approach offers numerous advantages, including improved scalability, enhanced flexibility, and greater fault tolerance, making it well-suited to the modern demands for efficient, reliable, and adaptable software solutions.
Amidst this shift, the importance of canary deployments in asynchronous processing becomes more pronounced, yet the industry lacks standardized solutions to effectively manage these challenges. This is where Worker Controller, a software developed from the ground up at Zomato, comes into play.
Worker Controller is a multi-tenant, high-available, push-based Kafka consumer & remote processor, designed for asynchronous, short-running jobs. Currently empowering hundreds of Kafka consumers across various microservices at Zomato.
It provides some of the key features for the management & resilience of async tasks.
In this breakout session, We'll delve into pressing issues in the Kafka worker systems, such as canary deployments, task prioritization, and retries. Learn how Worker Controller elegantly addresses these challenges.
We'll unpack the architecture of the Worker Controller, spotlighting its ability to seamlessly support multi-tenancy, elastic scaling & controlled consumption of Kafka topics for the resilience of downstream services. Then we will go into the performance of the system by comparing it with Envoy.