Kafka In the Cloud: Why It’s 10x Better With Confluent | Get free eBook

How Did We Move the Mountain? - Migrating 1 Trillion+ Messages Per Day Across Data Centers at PayPal

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications?

At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications.

In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.


Lei Huang

Lei Huang is a MTS2 engineer in PayPal who has extensive experiences in messaging and streaming infrastructure and platforms. She has been building and managing various large scale messaging systems including Kafka and ActiveMQ, and leading on key projects and initiatives in the PayPal messaging team.

Na Yang

Na Yang is an engineering lead at PayPal where she focuses on building highly reliable and scalable messaging and streaming platform. She has extensive experiences of building large-scale infrastructure, big data platform and distributed systems at PayPal and her previous companies MapR and Quova.