Confluent Cloud Q1 Launch: Build a Secure Shared Services Data Streaming Platform | Learn more

Sharing Data Among Microservices: How Change Data Capture with Kafka Connect Came to Our Rescue

Fully embracing the 鈥渙ne database per microservice鈥 principle can present challenges for the management of data across the whole ecosystem. Ideas like event sourcing, CQRS and applying Saga patterns approach the problem from different perspectives. Change Data Capture (CDC) is another method that can help stream changes made to a database into an event medium such as Kafka. When applied successfully, CDC keeps all other microservices and their databases informed.

At Brolly, we have implemented a log-based Change Data Capture (CDC) solution using Kafka Connect and Debezium. Having built and productionised the solution, we know what we have done well, what we could have improved and what challenges you may face designing your own.

We share the reasons behind our decision to use CDC, some insights about what to watch out for and lessons learned along the journey. We touch on topics such as how to guarantee transactional messaging by applying the famous outbox pattern, the implications of having badly designed aggregates, how Avro and schema registry can assist, what to do when processing a message on the receiving end fails for whatever reason, and how to mitigate the risk of processing an event multiple times or out of order.

Chinese Japanese Korean


Ali Nazemian
Milad Vadood