"Our hyper-personalized approach is delivering measurable results. In our A/B testing, we’ve seen a significant increase in customer conversion rates. That is proof that our decision to adopt a real-time event streaming approach was the right one."
Jon Vines
Engineering Lead, AO.com
AO is one of the UK’s leading electrical retailers, operating predominantly online for the last 20 years and selling over 9,000 electrical products to millions of customers across the UK and Germany. A business that not only prides itself on putting customers first, AO is just as dedicated to creating an amazing place for their 4,000 employees to not only work but thrive.
The COVID-19 pandemic caused a dramatic shift in consumer shopping habits, which led to a sharp increase in growth at AO. John Roberts, AO Founder and Chief Executive Officer, underscored the scale of the change when he explained that “the pandemic accelerated a shift in customer behavior towards online shopping–we saw 10 years’ change in consumer behavior in just 10 weeks.”
Helping to enable the rapid changes needed to support this surge in growth is a realtime event streaming platform based on Confluent and Apache Kafka®. This platform enables the AO Customer Personalization team to deliver hyper-personalized online experiences by combining historical customer data with clickstream data and other real-time digital signals from across the business. Just as importantly, it is enabling AO development teams to accelerate the rollout of new business capabilities.
“With data in motion and Confluent, we can deliver capabilities at pace, and do so with the governance and guardrails that we need to effectively scale the organization,” says Jon Vines, Engineering Lead at AO. “Pace became even more crucial during the pandemic because the world moved so rapidly from predominantly in-store shopping to online. The speed at which we are able to create new use cases that improve the customer journey with Confluent Cloud is helping us to cement our online market leadership position, even as we continue to adapt to ongoing changes.”
After the successful deployment of its first event streaming use case focused on hyperpersonalization, AO worked with Confluent Professional Services to progress rapidly in event streaming maturity, building to the point where reuse of data, efficiencies of scale, and the platform effect are reinforcing one another. “The speed at which teams are coming with new use cases and combining event streams from different sources is increasing, as is the speed at which we’re bringing teams onto the platform. We’re really at the cusp of the next phase, as our usage of event streaming and the value we see from it is starting to snowball and gain momentum every day,” says Vines. “Our initial efforts have been focused on online sales, but we’ll be using Confluent Cloud and event streams to improve operational efficiencies–including deliveries, for example–by integrating through layers and across the organization.”
Business Results
Customer experiences improved with real-time hyperpersonalization. “Our customers are happiest when we respond instantly to their individual needs,” says Vines. “With Confluent Cloud, we can create a single view of each customer, giving them what they want right in the moment, including product suggestions and relevant promotions to help guide their shopping decisions. That hyper-personalization is a huge differentiator for us and goes right to the heart of AO’s mission.”
Customer conversion rates increased by up to 30%. “Event streaming with Confluent Cloud is producing measurable results,” Vines explains. “In our A/B testing, we’ve seen a significant increase in customer conversion rates–up to 30%. That is proof that our decision to adopt a real-time event streaming approach was the right one. I expect even bigger benefits as we continue to grow our capabilities and expand into new use cases.”
Pace of innovation increased. “Confluent Cloud gives us the tools we need to drive innovation,” Vines says. “Once we made data available as an event stream via Confluent Cloud, we soon saw two or three other teams coming onboard to access that data for multiple use cases that weren’t even considered as part of the original plan. Those teams can rapidly achieve their goals in a decoupled way, that is, without creating new pointto-point integrations. As a result, we’re more agile and our teams can move much faster because they’re less dependent on other parts of the organization.”
Developers focused on value-add features, not operations. “Before Confluent Cloud, when we had a broker outage, developers would have to stop their development work and focus on operations until the problem was sorted out, which in some cases took up to three days,” Vine remembers. “With Confluent Cloud, we have a rock-solid production Kafka cluster that is fully supported and fully managed. Confluent takes care of everything for us, so our developers can focus on building new features and applications that add value to the business.”
Data at the speed of business. Integrated stock availability data to better guide customer journeys, informing of real-time stock updates and ensuring displayed offers for products are in stock.
Technical Solution
The AO team’s first steps with event streaming included extracting information from order processing and other systems using change data capture (CDC). The resulting streams were handled by a self-managed Kafka cluster hosted in AWS EC2 instances. This setup has since been replaced by Confluent Cloud, which now also handles clickstream events from AO web servers via raw topics and topics that have been enriched with additional customer context via the Kafka Streams API. The enriched topics feed the company’s backend services with events flowing back to the web server used to inject hyper-personalized data into the customer experience. “Running Confluent Cloud on AWS enables us to take advantage of the scalability of cloud-native approaches as we build our applications,” says Vines. “It also gives us the ability to decouple those applications and modernize where appropriate, bringing in data from sources like SQL Server and MongoDB while also expanding our cloud-native native footprint by integrating with services like Fargate, Lambda, and S3.”
Using the streams API, Confluent Cloud Schema Registry, and Kafka Streams API, such as S3 Sink, HTTP Sink, MSSQL source, and MongoDB source, have played a vital role in AO’s event streaming journey. Kafka Streams enables AO teams to transform and enrich event streams, while Schema Registry provides an evolvable data model to support effective data governance, and the connectors enable straightforward integration of Kafka to other systems. “The streams API is very important for us because it enables us to build up different views and create new stream processing applications,” says Vines. “And with Schema Registry, we get a clean separation between producers and consumers, so we can easily add new types of data without worrying about breaking existing applications.”
Seeking to build upon the momentum from its first event streaming use case, AO engaged Confluent Professional services to take advantage of the experience and SMEs to support them on their journey to set data in motion. “The success we saw with the original data streaming use case and the massive amount of growth we’ve seen recently led us to take a step back and think about using data in a more strategic way,” says Vines. “We wanted to build upon the value of our event platform as an asset for the organization, and that includes building the skills and expertise within our teams to use it effectively.”
A key objective of the project was helping more AO teams to realize benefits from–and contribute to–the event streaming platform. “One of the biggest goals for this initial engagement was a self-serve capability for teams, so that they can get on the platform independently, know how to find and use the data that’s already there, and create topics that are consistent with our overall approach to contribute data from their domains,” says Vines. “Governance and discoverability were a big part of that, because we want the topics to be consistent with our overall approach and strategy, and that includes everything from topic names to the type of data that’s available and whether it contains PII, for example.”
As part of the engagement, Confluent engineers delivered new tools to support metadata management, process management, and onboarding of teams to the event streaming platform. Based on a knowledge graph, this tool provides a link between platform and process management software. “This tool enables us to tackle topic creation in a much more controlled way. Instead of developers creating any topic they want, we now have a safer, better governed process,” says Vines. “With the tool, we can not only ensure that we create topics within the guidelines that we’ve established, but we can create them quickly. Instead of waiting around for a formal decision from someone who may not be available, we can deploy from our beta environment through to production in about 30 minutes.”
Further, the AO team was able to lean on the experience of the Confluent engineers to better guide customer journeys by adopting Kafka Streams to improve the customer experience by reducing latency. “We had one application that was using the Kafka Consumer API and Producer API to produce a view of the visitor session. Depending on the size of that session object, we started to see some latency,” says Vines. “By moving that functionality into a streaming application with Kafka Streams, we have a much quicker and consistent way to maintain state, reducing average latency and the number of database calls we were making significantly.”
Vines sums up the AO event streaming journey, saying, “We want to be brilliant for our customers, in the simplest and easiest way. We treat every customer like they’re our gran, by understanding our customers’ needs–and responding to them in real-time–we’re delivering on our promise. Confluent Cloud is a critical enabler for us, allowing us to treat each moment as a one-on-one opportunity to provide a great customer experience. And, we’re not done yet. The potential is almost limitless as we continue to learn and innovate.”
