Confluent Cloud Q1 Launch: Build a Secure Shared Services Data Streaming Platform | Learn more

Enforcing Schemas with Kafka Connect

Applying some measure of governance over how schemas are managed helps ensure good quality data, as well as better lineage tracking and governance. At Saxo, we have been on a journey to take control of how we manage our data through the use of rich, governed schemas. We hit a challenge when we wanted to ingest data with Kafka Connect, as there was no way to ensure the data coming through was matched with these existing schemas. We were left having to either build a second step of manual transformations for simply matching generic data into our internal schemas, or play a lengthy game of cat and mouse with Connect exceptions and complex per-field transformations. During this talk, we will be presenting how we tackled this issue by developing our own Schema Matching transformation. Our SMT can automatically match fields into a referenced schema. We will go through our experience designing the solution, and some of the key findings developing the SMT for both Avro and Protobuf.


David Navalho
Anatol Lutski