Sainsbury’s Revolutionises Its Supply Chain with Real-Time Data Streaming from Confluent

With over 1,400 stores nationwide, Sainsbury’s is the UK’s second largest supermarket chain. Operating since 1869, it’s become a household name and an icon of British retail. But legacy IT systems and data silos were stalling out initiatives to optimize the customer shopping experience, preventing the retailer from innovating as fast as it wanted to.

Sainsbury’s needed a system that could manage, store, and process enormous volumes of data in real time to jumpstart their digital transformation journey. Confluent provided the retailer with a fully-managed data streaming platform that was able to free up data from legacy systems and allow Sainsbury’s to transition to a truly cloud-native configuration.

Sainsbury’s is one of the pioneers of grocery retailing in the UK. The business first opened its doors in 1869, and later became an early adopter of the self-service shopping model that transformed the way people purchased food.

As the UK’s second-largest grocery retailer—serving millions of customers in over 1,400 stores nationwide—Sainsbury’s success hinges on the flexibility of its supply chain and logistics ecosystem, with real-time data being the critical enabler. Achieving the most optimal product availability is critical to profitability, customer experience, and retention. So, to support customer facing initiatives, improve operational analytics, and overhaul its supply chain’s digital systems, Sainsbury’s needed a modern data infrastructure that would transform its legacy architecture to provide real-time data streaming and analytics, opening new capabilities that guide powerful, continuous business insights and decision making.

Sainsbury’s started by leveraging open source Apache Kafka® for data streaming, but quickly upgraded to Confluent to handle the significant, complex data sets, and also to access premierlevel support where they needed it most. Sainsbury’s has already seen improved product delivery to stores, is better able to anticipate customer needs, and can even offer reduced prices to customers because of increased efficiency.

“We started out managing our own Kafka clusters, but as soon as we got to scale, we realised it would be too inefficient and expensive for us to keep self-managing. Thankfully, with Confluent, we get fully managed products and services that go beyond just infrastructure; they help us scale Kafka, handle upgrades, and solve problems when required.” —Pedro Baeta, Senior Engineering Manager at Sainsbury’s

The Challenge - Legacy Databases Were Slowing Down Innovation

Sainsbury’s was looking to completely revolutionise its supply chain. The company needed a solution that could handle huge volumes of event data, and a platform that could facilitate real-time data access and connections between multiple databases and logistics systems. The transformation also needed to be seamless and transparent, as Sainsbury’s operates 24/7, and any disruption could heavily impact retailers and customers.

Before deploying Confluent, it was difficult for Sainsbury’s to access the data it needed to drive new projects forward at pace. The retailer’s legacy systems were so essential to day-to-day operations that engineers were reluctant to touch them.

As Baeta explains, “Our customer-facing initiatives would stall due to lack of access to data. A lot of our data lived in very old legacy systems which were so critical for the running of the business that no one wanted to touch them because if they stop, the business stops.”

The Solution - Becoming Data and Cloud-Driven

As one key part of the investment in its supply chain, Sainsbury’s chose Confluent to form the data streaming backbone of its new digital transformation program and help with the transition to a cloud-native platform.

Confluent provided Sainsbury’s with a data pipeline that translates an action or real-world event like a purchase into a technical event, with both of them being connected instantly. It also allows the retailer to set event-based triggers, like when there is low stock on a product, and react in real-time. This results in more reliable product delivery in-store and better flexibility to adapt to unforeseen incidents.

Confluent also took over governance of the retailer’s Kafka services. By removing the operational burden, in-store teams now have the capacity to focus on what matters. Engineers can stay agile and dedicate their time to building business logic—accelerating the time to market for new initiatives—all while regaining time and reducing costs.

Sainsbury’s also benefited from pre-built, fully managed connectors and integrations with Schema Registry, which helped link all of their external systems—simplifying the development, deployment, and management of connectors, and helping the retailer to organize and interpret its data more intuitively.

For customers, this provides better availability of products, faster response times, and reduced prices thanks to efficiencies and cost savings further up the supply chain. With event-driven, real-time data, Confluent has helped Sainsbury’s forge a strong relationship between day-to-day business and technology.

Another key element of the project was security. In retail, any kind of data breach, disruption, or downtime can be devastating. To ensure Sainsbury’s new systems are resilient against future threats, Confluent provides interconnectivity between different cloud providers like AWS and Azure, adding an extra layer of security and disaster recovery on top. This means the retailer is protected if any cloud service experiences downtime, which greatly improves the strength and flexibility of its supply chain. With Confluent, Sainsbury’s now has seamless multi-cloud access, which is invaluable.

“In retail, everything is becoming data-driven. It’s no longer just a facet of IT, it’s quickly become essential to the entire business. Kafka has become a company-wide nervous system for us, and Confluent has really taken it to the next level.” —Pedro Baeta, Senior Engineering Manager at Sainsbury’s

The Results - A Supply Chain Transformed

After deploying Confluent and putting its data in motion, Sainsbury’s continues to achieve great results, including:

Better forecasting and supply chain management

With Confluent’s fully managed product offerings, Sainsbury’s now can concentrate on business logic, and has the resources to deploy more advanced techniques in-stream—like machine learning (ML) models that analyse data and fuel new predictive modelling for its forecasting engine. The retailer can now further enhance its ability to monitor and respond to ever-changing customer needs, and use Confluent’s data streaming to predict and prevent potential supply chain disruptions. This data access has already proved invaluable in helping Sainsbury’s tackle ongoing disruptions from external factors like Brexit and COVID-19.

Baeta explains, “We’ve acquired a SaaS forecasting ML solution being trained with three years of historical data, around 600 billion rows of records that are moving around alongside all-new real-time data streams. Confluent gave us the support we needed to implement this and gave us the tools to transform the data into actionable insights for our supply chain and commercial operations. This has already translated to cost-savings across the business, and will no doubt continue providing value for years to come,” says Baeta.

Real-time data streams

Sainsbury’s is now leveraging the power of fully managed ksqlDB, which builds upon Kafka Streams to provide an event-streaming database that can transform and process data in real-time. Through stream processing, Sainsbury’s can perform continuous computations over vast events streams and can filter, aggregate, and join the collections of data to get materialised views through SQL. This massive scale means Sainsbury’s can respond much more quickly to market shifts or other events.

“Real-time data events can be transmitted as technical events, and that is very powerful. When you design new systems, you can see what is impacted when you change the price of a product for example, and this creates a path for you to follow so you can connect the dots, and this simplifies the design process and creates a relationship between business and technology. This has more than doubled the efficiency of our engineering teams, as we can react to adverse events straight away,” says Baeta.

Easy connections with AWS

Confluent works in collaboration with Amazon Web Services (AWS), which means connecting to existing cloud systems is simple and intuitive. With Confluent, Sainsbury’s can easily connect with the AWS ecosystem with pre-built and fully managed integrations, which gives in-house developers more freedom and flexibility when building upon the platform.

As Baeta says, “One of the key advantages of Confluent is that it brings Kafka to the cloud such as AWS. Alternatively, you can also leverage Confluent to run Kafka on-prem and connect with multi and hybrid clouds.”

Additionally, Confluent’s integration with AWS helped to accelerate the onboarding process, as Sainsbury’s could transact directly in the AWS marketplace. This proved to be invaluable at the start of the project, as it helped the retailer effortlessly connect and deploy Confluent’s services with greater flexibility.

Ongoing support from the Confluent team

Sainsbury’s found that the support offered by the Professional Services and Education team at Confluent was essential, and helped to accelerate adoption, integrate best practices, and ultimately, decrease the time it took to become self-sufficient.

“One of the most critical things to do when implementing new technology at scale is making sure that the people around you understand the changes, especially when you’re dealing with a big paradigm shift and a more event-driven way of dealing with data,” Baeta says. “Confluent helped drive a new way of thinking by training our engineers, having people available to answer questions, and sanity-checking new designs. Their expertise has really made a difference.”

Confluent’s support also means that in-house engineers can now focus on what matters, instead of setting up databases or managing data access.

“As a result of our partnership with Confluent, we’ve regained an average of 20 hours per sprint, per team, as Confluent’s support means our engineers can now focus solely on business logic, instead of managing complex data clusters.” —Pedro Baeta, Senior Engineering Manager at Sainsbury’s

The Future - Data That Informs Decisions

Sainsbury’s can now react to situations in real-time, which is invaluable when it comes to problem-solving, building additional infrastructure, and accelerating engineering work.

“Collecting vast pools of data doesn’t drive innovation, it just slows progress down,” says Baeta. “You end up with vastly skilled people spending their time moving data around, instead of making it work. With Confluent, we can now react to situations and make decisions in real-time, which is critical for our future. And as for what’s next? Confluent to me is the ultimate accelerator in terms of engineering work, and we’re now in control of transforming the business in any way we choose.”

So, what does the future hold for the retailer? With Confluent’s help, Baeta and the rest of the Sainsbury’s team are on course to completely replace all of their outdated legacy systems, and implement a new supply chain that is flexible, adaptable, and future-proof.

“In three years, our business will be completely different, and we’re well on track to implement a new supply chain to deal with future challenges head-on,” Baeta says. “We’re looking forward to maintaining our current momentum and tackling issues like load flow efficiency, deliverables, and more. These are really complex problems and replacing these systems is very complicated, but with powerful technology and ongoing support from Confluent, we’re well on our way to making it a reality.”

Learn More About Sainsbury's