Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming

Presentation

High Available Task Scheduling Design using Kafka and Kafka Streams

« Kafka Summit APAC 2021

In any enterprise or cloud application, Task scheduling is a key requirement. A highly available and fault-tolerant task scheduling will help us to improve our business goals.

A classic task scheduling infrastructure is typically backed by databases. The instances/service that performs the scheduling, loads the task definitions from the database into memory and performs the task scheduling.

This kind of infrastructure creates issues like stateful services, inability to scale the services horizontally, being prone to frequent failures, etc., If the state of these kinds of services is not maintained well, it may lead to inconsistent and integrity issues.

To mitigate these issues, we will explore a high available and fault-tolerant task scheduling infrastructure using Kafka, Kafka Streams, and State Store.

Chinese Japanese Korean

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how