Delivering Cloud-Native Data Pipelines with Kafka Connect on Kubernetes

Getting data between systems, particularly at scale, is a common challenge faced by data engineers. Pipelines need to be reliable, flexible, and scalable, and without requiring us to write the same boilerplate code each time.

Kafka Connect is a framework that provides scalable & fault-tolerant integration between Apache Kafka and other systems. It can be deployed on containers making it easy to scale for increased capacity, throughput, and resilience.

We will give a short intro to Kafka Connect and container technologies before proceeding to a deep dive into practical applications. Attendees will learn about:

• Real-world Kafka Connect pipelines.
• How to build custom connector container images
• Configuration, and orchestration of Kafka Connect pipelines with Kubernetes using GitOps.