Unlock the full power of your event streams with Stream Governance | Learn More

Tales From the Four-comma Club: Managing Kafka as a Service at Salesforce

« Kafka Summit Americas 2021

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including:

How to manage multi-tenant clusters in a hybrid environment
High volume data pipelines with Mirus replicating data to Kafka and blob storage
Kafka Fault Injection Framework built on Trogdor and Kibosh
Automated recovery without data loss
Using Envoy as an SNI-routing Kafka gateway

We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Presenters

Lei Ye is a Principle Software Engineer at Salesforce focusing on infrastructure security, data lake and data processing. Lei Ye is a Principle Software Engineer at Salesforce focusing on infrastructure security, data lake and data processing.

Paul Davidson is a Software Architect at Salesforce, working on the team responsible for providing Kafka-as-a-Service across the organization. Paul was also the original developer of the open-source Mirus data replication tool, which Salesforce uses to provide high-volume cross-site data pipelines.