Build your real-time bridge to the cloud with Confluent Platform 7.0 and Cluster Linking | Read the blog

Simplifying Asynchronous Service Communication with Kafka, Debezium and the Outbox Pattern

Loggi is one of the Brazilian unicorns which is transforming logistics through technology to connect the entire country. At Loggi, we've widely used the CDC pattern with Debezium + Kafka to capture the changes from our main database as a way of communication between services, as an Event Notification pattern. Later, we concluded this wasn't enough, since we often need to join information from multiple tables to take an action. In this talk, we describe this journey, and how we leverage Debezium’s Outbox pattern to transactionally emit events following the Event-Carried State Transfer pattern, where all the needed information is sent to Kafka at once, therefore avoiding consumer requests to enrich the event data. Additionally, we describe an Event Broker service we’ve implemented to transform Kafka’s consumption from pull-based to push-based, helping other teams to integrate their services into this new architecture faster.

Presenter

Ernesto Matos

Ernesto is a Senior Software Engineer at Loggi. He works in the infrastructure team which, among other things, takes care of the company's data pipelines. He also holds a Ph.D. in computer science and researched software testing and formal verifications in the past.