Put a Topic on It

In 2018, we at T-Mobile for Business (TFB) were charged with developing an event-driven system where we would be receiving events from other organizations within T-Mobile through an in-house enterprise-wide message bus and then enhancing the data from those events with data coming from our own services.

The original assumption made by our architects was that we would be making limited use of Kafka. We immediately and successfully campaigned to dramatically expand our use of Kafka. Now we're reaping the benefits of this commitment we made, from the start, to Kafka. We have a resilient near-real-time streaming system that can gracefully respond to load and avoid disruptions to upstream systems; that can gracefully respond to failure and avoid disruptions to clients and downstream systems; and which gives us just the right level of control over which data flows where and when. Now we're seeing our organization continually shift more and more APIs to point at the databases our pipelines populate, in the process giving TFB's customers a more responsive experience.

In this talk, I'll explain what we call inbound and outbound Kafka topics and use those concepts as the launching pad to discuss:

• The importance of separating data capture from data processing.
• The power of Kafka as a circuit breaker.
• The usefulness of reusing the same pipelines, but not necessarily the same topics, for both event processing and data migrations/data fixes.
• When to do replays and when not to do replays, and the value of selective, configuration-based reprocessing.
• The three categories of failure you can encounter in a Kafka consumer and why one of them (normally) should not be treated as an exception.

Take this talk to your organization to make the case for Kafka, and for more Kafka, as you go about designing applications where your data is your lifeblood.