Elevate your stream processing w/ The Force of Kafka + Flink Awakens | Read Now

Presentation

Testing SMTs? Testcontainers to the Rescue!

« Kafka Summit London 2023

Kafka Connectors are ready-to-use components useful for importing and exporting data between Kafka topics and external systems such as databases or file systems. Along with the connectors, there are ready-made and easy-to-use Single Message Transformations (SMT) for making modifications to message values and keys. SMTs are relatively easy to build but testing them is another story. We faced several challenges in how to do a proper test because we couldn't faithfully replicate what could happen inside Kafka Connect when using Unit Tests. To make matters worse, we often got data in different formats or patterns than we expected, resulting in several errors while the SMT was being used in different environments.

Join us for a lightning session where we'll show you how to do proper integration testing for custom SMTs, using Testcontainers as the best, most accurate, and solid way to test these complex Kafka Connect components. In the end, we get an SMT that is more robust, cleaner, easier to debug and, most importantly, ensure it does what we expected when running it inside our Connect clusters.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how