Single Message Transforms (SMT) in Kafka Connect

Trendyol was established in 2010 to provide a seamless e-commerce experience to our customers and vendors. Trendyol sellers need to see their current account statement contains different kinds of items such as settlements, invoices, receipts, etc. in a single request and near real-time. However, according to our microservices architecture, all these items are stored in separate datasources as different domain objects. Collecting all these items from different sources causes performance issues. So we decided to migrate the specified data to a single datasource and keep the datasource up to date. We encountered some problems:

1. Migration of only a subset of data from different datasources
2. Keeping the data up to date 3- Handling the huge data size

In this session, we want to share how we solved them by using just Debezium and Kafka on the Confluent Platform. We used Debezium’s ability of getting an initial snapshot, Change Data Capture pattern, and particularly SMT.