Reliable Event Delivery in Apache Kafka Based on Retry Policy and Dead Letter Topics

In any sufficiently complex IT system, at some point, we realize that we have to answer the question: what if something goes wrong. We can do nothing and swallow the erroneous events. We can also stop processing and wait for the fix.

But what if we want some sensible retry policy, and in the absence of success - postponing events in a Dead Letter Queue (DLQ). It turns out that Kafka has no support for such a scenario. In this short presentation, I will talk about a pattern based on three topics: operational, retry, and DLQ, and how it can be handled programmatically.

Based on a real-world example of a messaging system created for one of the biggest European banks.