Elevating Kafka: Driving operational excellence with Albertsons + Forrester | Watch Webinar


Enforcing Schemas with Kafka Connect

ยซ Kafka Summit Europe 2021

Applying some measure of governance over how schemas are managed helps ensure good quality data, as well as better lineage tracking and governance. At Saxo, we have been on a journey to take control of how we manage our data through the use of rich, governed schemas. We hit a challenge when we wanted to ingest data with Kafka Connect, as there was no way to ensure the data coming through was matched with these existing schemas. We were left having to either build a second step of manual transformations for simply matching generic data into our internal schemas, or play a lengthy game of cat and mouse with Connect exceptions and complex per-field transformations. During this talk, we will be presenting how we tackled this issue by developing our own Schema Matching transformation. Our SMT can automatically match fields into a referenced schema. We will go through our experience designing the solution, and some of the key findings developing the SMT for both Avro and Protobuf.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how