"Confluent Cloud is the core infrastructure that is enabling us to realize our vision of having a single, always-up-to-date view of customer data and our vision of real-time self-optimizing journeys, which maximizes uplift by identifying the optimal campaign for each customer."
Shai Frank
VP of Product, Optimove
Optimove, the Science-First Relationship Marketing Hub, helps customer-centric businesses drive growth using its real-time customer data platform (CDP) that ingests customer data from multiple sources and transforms it autonomously into actionable insights at scale. Optimove combines predictive customer analytics, AI optimization technologies, and a multichannel campaign orchestration engine, enabling marketers to consistently target customers with the right message at the right time via the most effective channels.
With the recent acquisition of Axonite, Optimove has expanded the real-time data capabilities of its platform with an event streaming framework based on Confluent Cloud and Apache Kafka. This event-driven architecture is enabling Optimove to offer its clients real-time customer 360 data synchronization between the Optimove platform and customer systems, real-time self-optimizing journeys that take into account the predicted uplift of various campaigns for each customer, and event-based triggers that activate campaigns based on external events, such as a sold-out product being back in stock.
“Confluent Cloud is the core infrastructure that is enabling us to realize our vision of having a single, always-up-to-date view of customer data and our vision of real-time self-optimizing journeys, which use AI to maximize uplift by identifying the optimal campaign for each individual customer,” says Shai Frank, VP of Product at Optimove. “In the past, our view of the customer was mostly based on batch updates that we ran every 24 hours. Now, with Confluent Cloud and the ability to process tens of thousands of events every second, we are meeting the demands of the market to maintain a customer profile that is immediately updated with every purchase, support ticket, website visit, or any other customer interaction.”
With Confluent Cloud, Optimove is driving innovation that has already provided the company with a competitive advantage and is opening new avenues for future growth. “With the reactive infrastructure we’ve built using Confluent Cloud and the ability to query streams in real time with KSQL, we are better able to apply machine learning algorithms that optimize campaigns for our customers,” says Yuval Shefler, VP of Partnerships at Optimove. “KSQL is tremendously powerful for us because it enables us to be flexible with our data mapping in a way that many of our competitors cannot. Going forward, we plan to open this up to our customers, so they can bring in their own algorithms and execute their own queries.”
Business Results
New business opportunities opened. “As the person in charge of building our marketplace and our partnership ecosystem, being able to come to the table with an event-streaming system based on Kafka and Confluent is a big advantage,” says Shefler. “That’s because many of our prospects and customers are already using Kafka and Confluent. They trust Confluent with their sensitive data, and in our space that’s critical.”
Stress on external IT systems cut by 90%. “Using KSQL and Kafka Streams we have the ability to prepare data in complex structured formats for synching to external systems,” notes Adam Abrams, Data Streaming Group Manager at Optimove. “We’ve also created a Kafka Streams app that continuously compares the desired and actual state of the data in those systems, and computes an optimal set of API calls needed to perform an update. As a result, we’ve reduced the stress on those external systems by up to 90%, increasing scalability and reliability.”
Administrative overhead reduced by 17%. “We used to host and manage Kafka ourselves, but with Confluent Cloud we’re saving the equivalent of one person each month in administrative effort,” says Abrams. “In addition to increasing our focus on our core tasks, Confluent Cloud enabled us to set up multitenant clusters and better manage separate schema for our customers – and we’ll soon be taking advantage of the ability to easily scale up and scale down as needed.”
Technical Solution
In order to provide campaign orchestration and machine learning-based insights, the Optimove CDP must access real-time data from a wide variety of client systems, including customer relationship management (CRM) and point of sale (POS) platforms, as well as inventory management systems, loyalty programs, web back-ends, and disparate databases. Likewise, the hub must also connect to an array of campaign execution systems that provide push, email, direct mail, social media, and other services.
Optimove uses Confluent Cloud to solve the challenges of connecting to customer systems across the IT landscape to enable real-time event processing. “Confluent Cloud not only enables us to connect to multiple systems to collect customer data and customer activity events, it also lets us take multiple streams and join them in real-time, with Kafka Streams or KSQL,” says Shefler.
With its ability to perform real-time processing on Kafka streams, KSQL is central to the company’s solution. Shefler continues, “With KSQL we can offer our enterprise customers and third-party service providers an accessible tool that lets anyone write simple SQL queries in order to prepare the data themselves, without getting engineers or developers involved, helping them achieve shorter time-to-market while also supporting more adaptive data models.”
After developing much of the initial framework with Kafka Streams, the Optimove team is increasingly using KSQL for core functionality. This includes stateless transformations of delimited data into JSON or Avro formats, windowing, and other forms of data enrichment. It also includes using simple SQL queries to prepare data to be pushed into Zendesk, NoSQL databases, and other destination systems. “KSQL gives us tremendous flexibility in using our single unified version of customer data,” says Shefler.
The machine learning algorithms that are at the heart of Optimove self-optimizing journeys rely on state machines that the team manages with Kafka Streams as well as feature vectors that are aggregated in real-time from customer data. “With Confluent Cloud we track millions of customer journeys in real time, and whenever there is a state switch it can trigger an event on a topic that will in turn trigger additional actions automatically,” says Abrams.
The Optimove team is using the fully managed Confluent Cloud Schema Registry to help ensure data compatibility for their customers, and they are also integrating data flows with other cloud services including Google Cloud Pub/Sub and BigQuery. “From a technical perspective, Confluent really serves as a big part of the solution for us” Abrams concludes.
