Live Demo: Build Scalable Event-Driven Microservices with Confluent | Register Now
In today's fast-paced global E-Commerce industry, the amount of data generated by online shoppers is massive. To deliver real-time analytics, effective advertising campaigns and machine learning based personalized recommendations are crucial. However, building a reliable and scalable data pipeline to support this is a challenging task.
In this talk, we'll share how we tackled the challenge of building a fully managed robust data pipeline using a combination of streaming analytics, batch processing, data lake, and machine learning. Our platform, built on Google Cloud Platform and powered by Confluent Kafka, enables us to process a massive volume of events per day.
We'll dive into the technical details of our architecture, tech stack, and data flow, including how we use
• Kafka Streams Java applications which are deployed in kubernetes to consume, deduplicate, transform, filter, and write data into HBase NoSQL database for real-time analytics,
• Push to Meta for advertising campaigns,
• Google AI for personalized recommendations,
• Confluent sink connector to push events to Google Cloud Storage and BigQuery, and ksqlDB for bot filtering.
• We'll also cover our observability, monitoring, and deployment practices.
But we don't want to just talk about our pipeline, we want to help you build one too. You'll leave our talk with practical insights and lessons learned from our experience, including tips on building a reliable, fault-tolerant, and scalable data pipeline, choosing the right tech stack, and ensuring end-to-end observability. Join us, and learn how to take your data pipeline to the next level.