Safeguarding - Protecting Your Kafka from Misbehaving Clients

Kafka is a VERY flexible and configurable system. That flexibility has undoubtedly been key to its adoption, but it's also a double-edged sword. How can you ensure applications use cluster resources effectively, knowing that much configuration can be performed client-side? Hidden away from the watchful eye of operators. An incorrectly chosen value hidden in a config file somewhere can have drastic effects on application and Kafka cluster performance. At Conduktor, we have observed many examples of such issues reported by our customers. Common issues include producers with inefficiently configured batch size and linger, consumer groups with more members than subscribed partitions, and topics that unintentionally mix data with schema and without. In many of these cases, the impact of the issue is invisible to the client application but its impact on Kafka cluster performance is significant. Join us for a detective hunt as we discuss tools and metrics used to detect misconfigurations in clients, how to address them once discovered, and ways in which to ensure that new occurrences are prevented from arising in the future.