Kafka In the Cloud: Why It’s 10x Better With Confluent | Get free eBook

Moving 150 TB of Data Rsiliently on Kafka with Quorum Controller on Kubernetes

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.


Murali Kaundinya

Murali Kaundinya is a senior strategist with technology and architecture with extensive leadership and management consulting experience. He has served in a leadership role conceiving, executing and delivering transformational programs with Fortune 100 enterprises including financial services, health and life sciences, insurance and advanced technology. He was recently a Technology Fellow at Goldman Sachs where he transformed the firm’s distributed software engineering practices into a centrally managed platform optimizing on innovation, productivity, risk management and compliance. He has held leadership roles at Sun Microsystems, Inc. (now under Oracle) where he provided strategy consulting services to CXOs of Sun’s top clients across the world. Murali started his career at NASA/Goddard Space Flight Center, Greenbelt, Maryland. He has published and presented extensively on many technology topics. He holds several patents in RFID and in the field of Telemedicine.