Lessons Learnt Managing and Scaling Kafka: 1 Trillion Messages Per Day @PhonePe

Greetings! I am Nitish Goyal, a passionate Software Architect dedicated to pushing the boundaries of technology and innovation with 11 years of experience in the tech industry.

In my current role, I lead a dynamic team focused on building Platform/Infra services for PhonePe. Together, we've successfully built API-Gateway, RealTime Analytics Platform, Metrics Platform, Anomaly Detection Service, Authn/Authz Platform, Usage & Quota Platform and numerous services to support PhonePe hyper growth.

Prior to my current role, I made impactful contributions as a Software Developer at Sprinklr and CitiBank.

My expertise lies in managing and scaling large scale clusters, cost optimisations at hardware, and building horizontal services capable of handling high rps and use cases spanning across various products and teams.

I am a firm believer in continuous learning, holding a BE in Computer Engineering from Delhi College of Engineering and CORe diploma from Harvard University.

I'm a software engineer currently serving as an architect at PhonePe, a leading fintech company revolutionising digital payments and financial services in India. Within the data platform team, I spearhead the development of core infrastructure and platform services crucial for managing vast volumes of multi-modal data. My journey at PhonePe has been marked by a series of impactful projects spanning real-time streaming, online aggregations, user profiling, fraud detection, machine learning inference, and data warehousing. I'm deeply drawn to the intricate challenges of engineering, particularly those posed by large distributed systems. My expertise lies in navigating the delicate balance between consistency, fault tolerance, and performance in system design. Beyond the realm of technology, I'm an avid quizzer, non-fiction reader, and enthusiastic hiker.