Kafka Client State on Kubernetes: Challenges and Lessons

While Kubernetes is a powerful tool that solves a multitude of problems, it's not always the optimal choice for every application. Deploying Kafka clients with state on Kubernetes can be a complex undertaking, and careful consideration is required to determine if the benefits outweigh the challenges.

Utilizing a Kubernetes StatefulSet can be advantageous in maintaining stateful applications, preventing the need to rebuild costly states during each deployment. However, operations like migrating the app to a new Kubernetes cluster introduces additional complexities, particularly in handling historical data without the need for intensive reprocessing.

In this presentation, we'll delve into our experiences and lessons learned from running production Kafka clients and stream applications on StatefulSets in a Kubernetes environment within a demanding 24/7 business operation. We'll explore the challenges we encountered and the strategies we developed to overcome them, providing valuable insights for those considering a similar approach.