Freight Clusters: Up to 90% savings at GBps+ scale | Learn more
Getting started with a Kafka Consumer or Streams application is relatively straight forward, but having those clients be highly available and resilient in a real-world application where compute is becoming more ephemeral is another matter. Kubernetes is a popular place to deploy these workloads, making these challenges more accessible and prevalent than ever.
In this talk I will introduce a simple consumer implementation with a default configuration and discuss the KIPs and features that have been introduced over time to limit how the hostile world of cloud computing can impact your real-time consuming applications.
Once the Kafka Consumer configurations are under our belt, we can see how these same concepts are applied and augmented in Kafka Streams, and then cater for new concepts such as maintaining and restoring local state store data.
Throughout the talk I will show out-of-the box Kubernetes features and deployment configurations that harmonise with the Kafka clients and their configurations to achieve a highly available consumer or streams deployment.
If you have found your real-time streaming applications stopping the world through rebalancing, starting up slower than expected through a routine deployment, taking an age to restore state or over time found them to be less reliable as your platform engineers make your Kubernetes cluster more awesome, you will hopefully find something in this talk you could apply tomorrow.