Testing SMTs? Testcontainers to the Rescue!

Kafka Connectors are ready-to-use components useful for importing and exporting data between Kafka topics and external systems such as databases or file systems. Along with the connectors, there are ready-made and easy-to-use Single Message Transformations (SMT) for making modifications to message values and keys. SMTs are relatively easy to build but testing them is another story. We faced several challenges in how to do a proper test because we couldn't faithfully replicate what could happen inside Kafka Connect when using Unit Tests. To make matters worse, we often got data in different formats or patterns than we expected, resulting in several errors while the SMT was being used in different environments.

Join us for a lightning session where we'll show you how to do proper integration testing for custom SMTs, using Testcontainers as the best, most accurate, and solid way to test these complex Kafka Connect components. In the end, we get an SMT that is more robust, cleaner, easier to debug and, most importantly, ensure it does what we expected when running it inside our Connect clusters.