Découvrez de nombreuses nouveautés à l'occasion de notre lancement du 2e trimestre 2023 : moteur Kora Engine, règles de qualité des données, et bien plus encore | S'inscrire pour une démo

Designing a Feedback Loop for Event-Driven Data Sharing

¬ę Current 2022

In an integrated business environment where heterogeneous database technologies are deployed, Kafka Connect offers data sinks and sources that easily enable seamless integration, abstracting data exchange details from senders and receivers. However, challenges may arise as integration grows in complexity.

Recently, the Enterprise Business Information Systems division at Jet Propulsion Laboratory was tasked with delivering an event-driven data exchange between two of its major systems. Delivering this solution successfully required conquering complex data dependencies across tables and respecting business and atomicity requirements. However, enterprise data exchanges also require a robust feedback loop to properly identify, disseminate and remediate errors in the process to maintain data integrity and user trust.

In this talk, we will discuss how we overcame these challenges and delivered a fully automated and robust data exchange solution by extending Kafka Connect, leveraging ksqlDB streams/tables and aggregations, and developing custom microservices.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how