Deep Dive into Kafka Connect Protocol

Kafka Connect is the highway to our central nervous system for streaming data into Kafka. This highway has allowed us to connect hundreds of external services without the need to write code for each individual integration. Kafka Connect is based on the concept of having Connectors and tasks that do the work for us. There’s two fundamental questions that come up when it comes to our highway: • How is this highway made? How are connectors created/started? • How do we make sure there’s no traffic jams? Also known as, • How are connectors evenly distributed across nodes?

In this presentation, we will answer the above questions and cover the following topics: • Breaking down how a Connector/task is created • Breaking down each Kafka Connect protocol(eager, compatible, sessioned) • Walk through rebalances for each protocol • Pros/cons of each protocol