Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Handling eventual consistency in a transactional world

Change data capture (CDC) is a widely used solution to offload data in real time from legacy systems to Kafka in order to make it available to all the other downstream consumer applications. Despite other solutions CDC can in fact guarantee at the same time low latency and a very small footprint on the source system. However when data is moved from a relational database to a distributed stream platform what is gained in terms of throughput and latency is lost in terms of strong consistency and not all consumers are able to manage this loss by themselves. There are different upstream solutions that can be implemented to mitigate this problem preserving different levels of consistency.

In this talk we’ll:

  • see what is eventual consistency and where strong consistency is lost while moving data from a database to Kafka
  • describe different solutions to preserve consistency working at the source level (i.e. outbox pattern and call back pattern), working on Kafka topology or working on an external storage (i.e. integration hub)
  • analyze the pros and cons of all the presented solutions in terms of consistency guarantees and latency loss

Presenters

Matteo Cimini

Lead Software Engineer with 4+ years of experience in Data Science and Software Engineering. Experience in designing cloud-based MLOps platforms and large-scale, multi-tiered, event-driven, distributed software solutions to drive real-world actionable business insights.

Andrea Gioia

I’m the CTO at Quantyca an italian consulting firm specialized in data management and one of the co-founder of blindata.io a SaaS Data Governance Platform. I have more than 15 years of experience working with data. I started after graduating, working on BI and DWH projects, I then went through the various changes that impacted the world of data (big data, iot, cloud) to arrive today to work on data governance and AI projects. My work is not really changed, it has only become more challenging and fun. In this journey my passion for technologies and data has grown at every step. I strongly believe however that the practice of data engineering has become the bottleneck of the entire IT industry. Fortunately the centrality of the data in all realities is increasingly changing this situation, generating a revolution in the field of data management that will be evident in the coming years. Let's roll. The best is yet to come!