Tales From the Four-comma Club: Managing Kafka as a Service at Salesforce

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including:

• How to manage multi-tenant clusters in a hybrid environment
• High volume data pipelines with Mirus replicating data to Kafka and blob storage
• Kafka Fault Injection Framework built on Trogdor and Kibosh
• Automated recovery without data loss
• Using Envoy as an SNI-routing Kafka gateway

We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.