Did you ever wonder why you need to configure advertised listeners for your Kafka brokers? How can clients be forward and backward compatible with brokers? Or how the consumer distributes the load amongst worker nodes in a cluster? Then this talk is for you!
Assuming no prior knowledge, we’ll look at how the Apache Kafka producer and consumer clients work from a protocol point of view. We’ll cover how clients bootstrap their connection to Kafka as well as the basic protocols that support records being produced to, and consumed from, Kafka. We will then go on to look at more advanced interactions like consumer group coordination and transactions.
Following this talk you’ll know how the Kafka client protocols work in detail and be able to tell your leaders from coordinators! The next time you have a problem you will not only be able to debug it more easily but also understand how to best utilize the Kafka protocol for your applications.