Connecting Offsets, Fault Tolerance, and Delivery Semantics in Kafka Connect

Offsets are ubiquitous in streaming data systems. The simplest analogy to understand offsets would be a video game where checkpoints allow you to avoid respawning from the beginning if your main character (Kafka Consumer, Connector, Streams application etc.) dies at any point. However, as is always the case with distributed systems, there is a lot more going on under the hood.

During this session, we will explore how offsets are represented, stored and used for source and sink Kafka connectors. We will also discuss how fault tolerance is achieved in Kafka Connect. This will be followed by a deep dive on delivery semantics in Kafka Connect and why they’re intrinsically linked with offsets - including how and when we can achieve the holy grail of exactly-once delivery semantics.

Audience members can expect to learn about the nitty-gritty details of a distributed system like Kafka Connect and the patterns that allow us to accomplish complex goals like fault tolerance and exactly-once delivery semantics which are essential to building robust data pipelines.