The Why Behind the How: Kafka Reliability from First Principles

Apache Kafka is known to be a highly-available system which stores and serves data reliably. Being a distributed system there is no one-config-fits-all and different use-cases require different trade-offs. Many talks cover ""how” to configure Kafka for resilience, but lack to explain the ""why"" behind the configs. Understanding configurations and their interactions from first principles breaks these chains, and unlocks Kafka’s full potential. In this talk we will touch on the how, but mainly peek behind the curtains to explain the why.

We will revisit the relevant parameters for brokers, topics, producers, and consumers that allows you to configure Kafka with different reliability and availability properties. We will dig into the glory details of these parameters and their interactions with each other, using building blocks that apply to distributed systems. This knowledge will enable you to understand and reason about the numerous available trade-offs, allowing you to configure Kafka for your use-case. Say goodbye to searching for “tutorial X” that matches your use-case; you might not find one anyway.