Introducing Streamhouse: the open data architecture for AI | Learn More
Scheduling workloads in real time distributed stream processing systems like Connect and Flink presents several unique challenges like optimal matching of resource availability with dynamic demand, efficient rebalancing of workloads and dynamically scaling of clusters. While there is no dearth of static and dynamic scheduling algorithms for these use cases, optimizing resource usage across heterogeneous workloads for multi-tenant clusters adds to additional complexity of these systems.
This talk goes over task assignment strategies used in Kafka Connect and Flink. We will dig into Incremental cooperative assignment used in Kafka Connect and contrast it with commonly used strategies in Flink such as static allocation, dynamic allocation, fine grained slot sharing/resource limits etc., We further go over considerations for picking the right scheduling strategy for various workloads and key criteria for evaluating these options.
Attendees will walk away with learnings of resource aware scheduling strategies in Kafka Connect, Flink and other streaming systems, enabling them to optimize resource utilization, improve performance, and ensure the stability of their streaming applications.