Getting data between systems, particularly at scale, is a common challenge faced by data engineers. Pipelines need to be reliable, flexible, and scalable, and without requiring us to write the same boilerplate code each time.
Kafka Connect is a framework that provides scalable & fault-tolerant integration between Apache Kafka and other systems. It can be deployed on containers making it easy to scale for increased capacity, throughput, and resilience.
We will give a short intro to Kafka Connect and container technologies before proceeding to a deep dive into practical applications. Attendees will learn about: