Configuring Kafka Connect To Be Successful At Scale

With Kafka Connect providing the ability to connect numerous external systems without having to write custom code it can be an integral part of the pipeline for putting data in motion. However, there are some settings and practices that can lead to a better experience if you are an admin of a Kafka Connect cluster as more teams add additional connectors for other use cases or more load is added to the Kafka Connect cluster overall.

This presentation will look at the best practices to configure Kafka Connect to output important data when evaluating if more resources are needed for a Kafka Connect worker or if a new node should be added to the cluster overall.

Areas of interest to be discussed:

• What memory areas make up a Kafka Connect process (JVM process)?
• What is verbose gc and why should it be enabled?
• Is Kafka Connect configured to give me the specific information for my connectors and their respective tasks?
• Which connect.protocol should I be using in environment?