Key Metrics To Uncover the Root Cause of Kafka Performance Anomalies

Kafka makes it possible to ingest and process large amounts of data concurrently by decoupling your data streams. This successful approach introduces some challenges when monitoring Kafka pipelines - to understand what’s happening, we need to monitor every single component and how they interact with each other.

In this talk, we will take a close look at Kafka’s architecture as well as the key infrastructure, JVM, and system metrics you should monitor for each of its components. Then, we will walk through how to diagnose common Kafka performance anomalies through observing patterns in the metrics for the various components. Finally, we will walk through setting up an open source observability pipeline with OpenTelemetry to enable you to collect and process Kafka metrics at scale.