[ウェビナー] ストリーミングデータメッシュを構築する方法 | 今すぐ登録

Presentation

Single Message Transforms (SMT) in Kafka Connect

« Kafka Summit Europe 2021

Trendyol was established in 2010 to provide a seamless e-commerce experience to our customers and vendors. Trendyol sellers need to see their current account statement contains different kinds of items such as settlements, invoices, receipts, etc. in a single request and near real-time. However, according to our microservices architecture, all these items are stored in separate datasources as different domain objects. Collecting all these items from different sources causes performance issues. So we decided to migrate the specified data to a single datasource and keep the datasource up to date. We encountered some problems:

  1. Migration of only a subset of data from different datasources
  2. Keeping the data up to date 3- Handling the huge data size

In this session, we want to share how we solved them by using just Debezium and Kafka on the Confluent Platform. We used Debezium’s ability of getting an initial snapshot, Change Data Capture pattern, and particularly SMT.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how