Fully-Managed, Multi-Tenant Kafka Clusters: Tips, Tricks, and Tools

Running a multi-tenant Kafka platform designed for the enterprise can be challenging. You need to manage and plan for data growth, support an ever-increasing number of use cases, and ensure your developers can be productive with the latest tools in the Apache Kafka ecosystem — all while maintaining the stability and performance of Kafka itself.

At Bloomberg, we run a fully-managed, multi-tenant Kafka platform that is used by developers across the enterprise. The variety of use cases for Kafka leads to bursty workloads, latency-sensitive workloads, and topologies where partitions are fanned out across hundreds or thousands of consumer groups running side-by-side in the same cluster.

In this talk, we will give a brief overview of our platform and share some of our experiences and tools for running multi-tenant stretched clusters, managing data growth with compression, and mitigating the impact of various application patterns on shared clusters.