Introducing Streamhouse: the open data architecture for AI | Learn More
Trust is essential with data, especially in financial streaming platforms, when users need assurance through auditing that there is no loss in data when it’s arriving from the source database to the data sink. Knowing that data is accurate helps with more confident and informed decision making.
In this talk you will learn from our experiences using Kafka, Debezium in detecting data loss in real time data pipelines, and retrieving lost data when data is streaming thereby improving both accuracy and trust of your data.
We will learn this through a live demo where we’ll:
Dive into the basics of change data capture and capture live changes from databases
Extract metadata from two end points (replication slot and Kafka connect) and find out how to detect if data has been lost
Correct this data loss by using incremental snapshotting to retrieve the lost data
By the end of this talk you will learn how to ingest live changes from databases using Debezium and Kafka connect, detect data loss while it’s streaming and learn to use incremental snapshotting to retrieve any lost data.