With Kafka Connect providing the ability to connect numerous external systems without having to write custom code it can be an integral part of the pipeline for putting data in motion. However, there are some settings and practices that can lead to a better experience if you are an admin of a Kafka Connect cluster as more teams add additional connectors for other use cases or more load is added to the Kafka Connect cluster overall.
This presentation will look at the best practices to configure Kafka Connect to output important data when evaluating if more resources are needed for a Kafka Connect worker or if a new node should be added to the cluster overall.
Areas of interest to be discussed:
- What memory areas make up a Kafka Connect process (JVM process)?
- What is verbose gc and why should it be enabled?
- Is Kafka Connect configured to give me the specific information for my connectors and their respective tasks?
- Which connect.protocol should I be using in environment?
Presenter
Travis Sweet
ConfluentI started working in support back in 2008 at IBM providing Level 2 Support for WebSphere Application Server on the WASSDK Team. After 5+ years I joined CloudBees and provided support Jenkins, Jenkins plugins, and Jenkins Enterprise plugins. After nearly five years I joined Confluent in 2018 have been putting data into motion ever since. I am currently focused on providing support for Kafka Connect and Kafka Connect plugins.