Build your real-time bridge to the cloud with Confluent Platform 7.0 and Cluster Linking | Read the blog

Delivering Cloud-Native Data Pipelines with Kafka Connect on Kubernetes

« Kafka Summit Europe 2021

Getting data between systems, particularly at scale, is a common challenge faced by data engineers. Pipelines need to be reliable, flexible, and scalable, and without requiring us to write the same boilerplate code each time.

Kafka Connect is a framework that provides scalable & fault-tolerant integration between Apache Kafka and other systems. It can be deployed on containers making it easy to scale for increased capacity, throughput, and resilience.

We will give a short intro to Kafka Connect and container technologies before proceeding to a deep dive into practical applications. Attendees will learn about:

Real-world Kafka Connect pipelines.
How to build custom connector container images
Configuration, and orchestration of Kafka Connect pipelines with Kubernetes using GitOps.

Presenter

Viktor Gamov is a developer advocate at Confluent, the company that makes an event streaming platform based on Apache Kafka. Back in his consultancy days, Viktor developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping architects and developers design and develop low-latency, scalable, and highly available distributed systems. He is a professional conference speaker on distributed systems, streaming data, JVM, and DevOps, and he regularly speaks at events like JavaOne, Devoxx, OSCON, and QCon. He co-authored O’Reilly’s Enterprise Web Development and writes on the Confluent blog.