[Demo+Webinar] New Product Updates to Make Serverless Flink a Developer’s Best Friend | Watch Now

Presentation

Sharing Data Among Microservices: How Change Data Capture with Kafka Connect Came to Our Rescue

« Kafka Summit APAC 2021

Fully embracing the “one database per microservice” principle can present challenges for the management of data across the whole ecosystem. Ideas like event sourcing, CQRS and applying Saga patterns approach the problem from different perspectives. Change Data Capture (CDC) is another method that can help stream changes made to a database into an event medium such as Kafka. When applied successfully, CDC keeps all other microservices and their databases informed.

At Brolly, we have implemented a log-based Change Data Capture (CDC) solution using Kafka Connect and Debezium. Having built and productionised the solution, we know what we have done well, what we could have improved and what challenges you may face designing your own.

We share the reasons behind our decision to use CDC, some insights about what to watch out for and lessons learned along the journey. We touch on topics such as how to guarantee transactional messaging by applying the famous outbox pattern, the implications of having badly designed aggregates, how Avro and schema registry can assist, what to do when processing a message on the receiving end fails for whatever reason, and how to mitigate the risk of processing an event multiple times or out of order.

Chinese Japanese Korean

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how