Bâtissez votre pont vers le cloud en temps réel avec la plateforme Confluent 7.0 et Cluster Linking | Lire le blog

Enforcing Schemas with Kafka Connect

Applying some measure of governance over how schemas are managed helps ensure good quality data, as well as better lineage tracking and governance. At Saxo, we have been on a journey to take control of how we manage our data through the use of rich, governed schemas. We hit a challenge when we wanted to ingest data with Kafka Connect, as there was no way to ensure the data coming through was matched with these existing schemas. We were left having to either build a second step of manual transformations for simply matching generic data into our internal schemas, or play a lengthy game of cat and mouse with Connect exceptions and complex per-field transformations. During this talk, we will be presenting how we tackled this issue by developing our own Schema Matching transformation. Our SMT can automatically match fields into a referenced schema. We will go through our experience designing the solution, and some of the key findings developing the SMT for both Avro and Protobuf.

Présentateurs

David Navalho
Anatol Lutski