Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Reliable Event Delivery in Apache Kafka Based on Retry Policy and Dead Letter Topics

In any sufficiently complex IT system, at some point, we realize that we have to answer the question: what if something goes wrong. We can do nothing and swallow the erroneous events. We can also stop processing and wait for the fix.

But what if we want some sensible retry policy, and in the absence of success - postponing events in a Dead Letter Queue (DLQ). It turns out that Kafka has no support for such a scenario. In this short presentation, I will talk about a pattern based on three topics: operational, retry, and DLQ, and how it can be handled programmatically.

Based on a real-world example of a messaging system created for one of the biggest European banks.

Presenter

Jacek Grobelny

Jacek is best in the depths of the IT systems backend. He likes to solve problems especially when the memes begin to develop. Married with two children. In his spare time he absorbs, though not uncritically, all manifestations of pop culture with a particular weakness for video games.