[Demo+Webinar] New Product Updates to Make Serverless Flink a Developer’s Best Friend | Watch Now
Our everyday digital experiences are in the midst of a revolution. Customers increasingly expect their online experiences to be interactive, immersive, and real time by default. The need to satisfy user expectations is driving the exponential growth of event-driven architectures in organizations of all shapes and sizes. And by enabling users to have real-time experiences whenever and wherever they want, 24/7, mobile drives this change further and faster.
This blog post demonstrates a simple way to build a dependable real-time betting experience by combining the power of Confluent Cloud with the capabilities of the Ably edge messaging platform. A betting use case is used for this demo, but the insights shared are relevant for any event-driven system that needs to process and stream real-time data at elastic scale over the internet.
The gaming and betting industry is an excellent example of why real time is critical for staying competitive. The need to keep users engaged (and ultimately to drive more revenue from bets) has led to the emergence of in-play betting. Odds are dynamically updated so users can place bets during a live sporting event, even on time-limited occurrences such as the outcome of a penalty. However, to stay engaged and become advocates of a platform, users need to trust it, and trust they will have a good betting experience every time.
Getting data to and from users across the internet in real time is a complex challenge. Failing to do it right leads to user dissatisfaction, as demonstrated by these publicly available reviews of various betting apps:
Customers leave platforms that don’t provide a reliable experience:
Delays, and even a single missed update, can significantly reduce user trust, resulting in the loss of customers. Therefore, it is critical that companies ensure the flow of data to and from users (mobile or otherwise) remains fast and reliable.
Apache Kafka® is becoming an industry standard for building secure, highly scalable, and reliable betting platforms that process millions of bets in real time. But it is only one part of the solution. Kafka is designed to operate in the backend, but you also need a way to connect your high-performance Kafka-centric pipeline to end users at the network edge.
A real-time betting solution has to deal with a rapidly changing and sometimes unpredictable number of end users. Large connection spikes are common, especially during events such as the Super Bowl, the World Cup, or a Formula 1 Grand Prix, which attract millions of simultaneous bettors. These users need to be able to place bets and receive updates (new odds, latest scores) in real time. After all, when it comes to live sports, even a single second can make a huge difference.
A dependable betting solution must have the following properties:
Together, Confluent Cloud and Ably easily provide all of these properties. Thanks to its elasticity, high availability, and reliability, Confluent Cloud is an excellent choice for building interactive betting systems—as showcased at the 2021 Kafka Summit.
Ably is an edge messaging platform designed to distribute and collect low-latency messages to and from web, mobile, and IoT devices, via a fault-tolerant, autoscaling global edge network. Ably is often used as a Kafka extension for sending mission-critical data to and from end users at the network edge in real time.
The newly released Ably Kafka Connector provides seamless no-code integration between Kafka or Confluent and Ably. The connector enables data processed at sub-second latencies in your Confluent ecosystem to travel to client devices in a fast and dependable way, with Ably as a message broker in the middle. Additionally, Ably makes it easy to stream event data generated by end users back into your Kafka deployments at any scale with the Firehose service.
The following architecture demonstrates how Confluent Cloud and Ably work together when engineering real-time betting functionality:
Whenever new odds are available, they are sent from Confluent Cloud to Ably via the Ably Kafka Connector. Ably then distributes the odds in real time to any number of Android app users over pub/sub channels.
In the Android app, users can see odds change in real time, and they can place their bets. Since Ably uses event-driven, bidirectional WebSockets as the primary transport protocol, a user not only receives odds, but they are also able to quickly place their bets over the same connection.
Ably streams the firehose of bets made into Confluent Cloud. ksqlDB is then used to create a materialized view of the current state of the order book, and to push updated odds into the Odds topic. Ably also built a Python microservice that queries ksqlDB to send personalized notifications to users, making them aware of various events: a win, a loss, or a new race starting soon (more about the microservice later in this article).
From here, the process covered so far repeats—there’s a continuous loop of data being processed and traveling in real time between the backend and the Android app, with Ably sitting as an edge messaging broker between Kafka and end users.
A betting solution built with Confluent Cloud and Ably offers the following advantages:
Before jumping into the step-by-step walkthrough, the following main components are used to build the real-time betting demo:
Let’s now dive into more details, covering how to set up each of these components, and demonstrating how they interact with each other. All of this is presented in the video below.
.
The rest of this article largely covers the same aspects presented in the video above.
Setting up the Confluent ecosystem is straightforward and involves a few simple steps. If you don’t have one yet, start by signing up for a Confluent Cloud account.
Once you’ve logged in to Confluent Cloud, create a Kafka cluster (default settings are OK for the purpose of this demo). Note that Confluent gives you the flexibility to select the cloud provider and region of your choice to deploy the Kafka cluster. In complementarity, Ably is underpinned by a globally distributed edge network that guarantees clients are always routed to the Ably data center that is closest to them.
Next, provision two Kafka topics, which will be used to store bets made by users, and new odds that are generated whenever bets are placed. By default, each Kafka topic has six partitions; you don’t really need all the six partitions for this demo, but in a real-life scenario, adding more partitions is the primary way to increase Kafka capacity so you can deal with high load.
Now that you have a Kafka cluster and topics up and running, set up a basic ksqlDB application. This demo uses ksqlDB to create a materialized view of incoming bets, and, based on that, generate new odds.
Similar to Confluent, getting the ball rolling with Ably is simple. Start by signing up for an Ably account.
Next, log in to your Ably account dashboard and create an Ably app.
From the Integrations tab of my Ably app, a Kafka rule is created, which allows you to stream the bets placed in the Android app into the Bets topic in Kafka.
Note that the Kafka rule is a new feature, and it’s currently in developer preview. It’s also an enterprise-only feature. However, if you plan to build this demo yourself, contact us, and we’ll talk about making the Kafka rule available to you for testing.
For data distribution to and from the edge, Ably uses channels, which are analogous to Kafka topics. However, Ably channels are provisioned dynamically at the point of use (there’s no need to pre-provision them ahead of time), and they have hierarchical namespaces to enable rule-based processing of messages (e.g., push notifications, firehose rules).
The Ably Kafka Connector is available on Confluent Hub. You can use it with Confluent Platform, or run it locally with Docker. Installing it is a simple process—follow the GitHub documentation for details.
Grab the code from the GitHub repo and run the project with Android Studio.
Quickly looking at the code, the following snippet initializes the real-time connection to Ably:
private void initAbly() throws AblyException { rtConnection = new AblyRealtime("Your Key Here"); Map<String, String> params = new HashMap<>(); params.put("delta", "vcdiff"); ChannelOptions options = new ChannelOptions; options.params = params; Channel channel = rtConnection.channels.get("outbound:odds:event1",options); channel.subscribe(new Channel.MessageListener() {
It also subscribes to the odds channel with in-channel message delta compression enabled. Delta compression helps massively reduce bandwidth and latency for last-mile data distribution by sending only the changes from the previous message to subscribers each time there’s an update, instead of the entire message. This compression is entirely transparent to the user, and the Ably library handles any missing data, and guarantees that the end user always gets a complete message.
Note that for testing push notifications, you will have to create a Firebase Cloud Messaging account, and set up Ably and Firebase accordingly—read the tutorial for details.
When you open the Android app, you should see the following:
By clicking on the Activate Device and Deactivate Device buttons, you subscribe and unsubscribe from notifications (which are delivered via Ably’s managed push notifications service). Push notifications may be sent to announce events such as a win, a loss, or a new race starting soon.
Due to push notifications being occasional batch jobs, they are less well-suited to a direct streaming integration between Confluent and Ably. So this demo uses a Python microservice that queries ksqlDB via REST, and sends push notifications through Ably. You can find the Python microservice in this demo’s GitHub repo.
Once you’ve clicked the Betting view button, you will be taken to the second screen, where you can see a list of cars with changing odds that you can bet on. To place a bet, just click on any of the car tiles.
Now if you go to the Dev console tab of the Ably app you created earlier, you should see the list of bets placed updating in real time.
Each of these events contains information about when the bet was placed, and the odds at that time. This allows the system to verify that any bet is placed with acceptable odds. Payment processing is not demonstrated here, but in a complete system, this would allow you to accept late bets if the risk isn’t too high.
All bets are streamed from Ably into the Bets topic in Confluent Cloud; this is achieved via the Kafka rule mentioned earlier. Bets are then consumed by ksqlDB, as shown below:
As mentioned before, ksqlDB is used to create a materialized view of the bets placed, and to push updated odds into the Odds topic in Confluent Cloud. With the help of the Ably Kafka Connector, the odds are then sent to Ably, and from Ably to the Android app, via an Ably channel.
This concludes the demo—we hope you’ll find it useful!
So far, this article has demonstrated the feasibility of using Confluent Cloud and Ably to power real-time betting experiences. But the demo should also serve as inspiration for building scalable, production-ready systems. A realistic high-level architecture for a real-time betting solution built with Confluent Cloud and Ably might look something like this:
Through sink and source connectors, Confluent Cloud makes it easy to add various additional components to your backend, such as databases. Confluent Cloud enables you to effortlessly scale to as many topics as you need, each one storing specific event data (like the list of fixtures), and using Schema Registry to safeguard data formatting.
ksqlDB allows you to create materialized views, and continuously transform, enrich, join together, and aggregate events. You can have various microservices (like the Python one used in the demo for sending push notifications) querying, consuming, and acting on ksqlDB data.
Ably offers 25+ client SDKs, targeting every major web and mobile platform. This means that almost regardless of what technology you are using on the client side, Ably can help intermediate the flow of data between the Confluent Cloud backend and end users. For example, besides Android, you can use Ably to stream data to and from iOS users and web pages.
In addition to powering real-time communication for sports enthusiasts placing bets, Confluent Cloud and Ably can be used to enable capabilities relevant to betting providers, such as risk management, fraud detection, and real-time monitoring.
We hope this post has helped you understand the benefits of combining Confluent Cloud and Ably when you want to engineer low-latency, scalable, and reliable digital betting experiences for your customers. Together, Confluent and Ably enable you to stream and process high volumes of data in your backend Kafka pipeline, and then broadcast messages in milliseconds to very high and quickly changing numbers of client devices at the network edge, in a fault-tolerant manner, without any update being delivered in the wrong order or more than once.
Although the PoC covered in this blog post relates to gaming and betting, you can easily combine Confluent Cloud and Ably to deliver scalable real-time digital experiences for a wide variety of use cases and industries, from real-time asset tracking and virtual events to healthcare and EdTech. In any scenario where time-sensitive data needs to be processed and must flow between the data center and client devices at the network edge in (milli)seconds, Confluent Cloud and Ably can help.
Confluent Cloud and Ably are fully managed, serverless solutions, abstracting away the massive infrastructure and engineering overhead and the associated financial costs required to build and manage your own proprietary real-time betting system capable of serving a global user base.
Getting started with Confluent Cloud and Ably is simple—both are almost no-code solutions, and you can be up and running with each in minutes. Additionally, integrating Confluent and Ably is a seamless and quick process made possible by the Ably Kafka Connector—try it out and see for yourself! You can also read more about how Ably helps you effortlessly and reliably extend your Kafka pipeline to end users at the edge.
If you’re ready to get started with Confluent Cloud, sign up for a free trial and use the code CL60BLOG for an additional $60 of free usage.*
Skai completely revamped its interactive, ad-campaign dashboard by adding Apache Kafka and an in-memory database—eventually moving the solution to Confluent Cloud. Once on the Cloud, they devised an ingenious architecture for reducing the number of topics they needed.
Turning events into outcomes at scale is not easy! It starts with knowing what events are actually meaningful to your business or customer’s journey and capturing them. At Confluent, we have a good sense of what these critical events or moments are.