Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evernorth Health Services

In this talk, we look at textbook examples of using kafka at scale. Specifically focused on Evernorth Health Service's journey of implementing microservices data pipelines, we provide an overview of the patterns we used while implementing CDC data pipelines for these micro services using confluent kafka, the challenges we faced, lessons learned and the unique solutions that we developed over the years to overcome those challenges. We also look at how we are moving these micro services eco systems to public cloud (AWS) and the strategies that we have/are implementing to ensure a smooth consumer cutover. We peek into how kafka consumers can cutover to a replicated topic using offsets based on create time timestamps and how/why this is critical for a downtime-less and data-loss-less consumer cutover for streaming consumers. To conclude, we take a look at how we are re-imagining these pipelines on AWS and how SaaS offerings like Confluent Cloud and Confluent connectors could play a major role.