This blog post was first published on the Confluent blog.
Our everyday digital experiences are in the midst of a revolution. Customers increasingly expect their online experiences to be interactive, immersive, and realtime by default. The need to satisfy user expectations is driving the exponential growth of event-driven architectures in organizations of all shapes and sizes. And by enabling users to have realtime experiences whenever and wherever they want, 24/7, mobile drives this change further and faster.
In this post, we will demonstrate a simple way to build a dependable realtime betting experience, by combining the power of Confluent Cloud with the capabilities of the Ably edge messaging platform. We have used a betting use case for this demo, but the insights we share are relevant for any event-driven system that needs to process and stream realtime data at elastic scale over the internet.
The realtime sports betting use case
The gaming and betting industry is an excellent example of why realtime is critical for staying competitive. The need to keep users engaged (and ultimately to drive more revenue from bets) has led to the emergence of in-play betting. Odds are dynamically updated so users can place bets during a live sporting event, even on time-limited occurrences such as the outcome of a penalty. However, to stay engaged and become advocates of a platform, users need to trust it, and to trust they will have a good betting experience every time.
Getting data to and from users across the internet in realtime is a complex challenge. Failing to do it right leads to user dissatisfaction, as demonstrated by these publicly available reviews of various betting apps:
Customers leave platforms that don't provide a reliable experience:
Delays, and even a single missed update can significantly reduce user trust, resulting in the loss of customers. Therefore, it is critical that companies ensure the flow of data to and from users (mobile or otherwise) remains fast and reliable.
Kafka is becoming an industry standard for building secure, highly scalable, and reliable betting platforms that process millions of bets in real time. But it is only one part of the solution. Kafka is designed to operate in the backend, but you also need a way to connect your high-performance Kafka-centric pipeline to end-users at the network edge.
Requirements for a dependable realtime betting solution
A realtime betting solution has to deal with a rapidly changing and sometimes unpredictable number of end-users. Large connection spikes are common - especially during events such as the Super Bowl, the World Cup, or a Formula 1 Grand Prix, which attract millions of simultaneous bettors. These users need to be able to place bets and receive updates (new odds, latest scores) in real time. After all, when it comes to live sports, even a single second can make a huge difference.
A dependable betting solution must have the following properties:
- High availability and fault tolerance - so it can continue to operate without interruption even in adverse conditions like infrastructure failures.
- Low latency - not only to ensure a good user experience, but also to minimize the risk of incurring high liabilities - the faster you receive and process bets, the quicker you’re able to generate new, updated odds and send them to users.
- Scalability - the system will often have to sustain a very high number of concurrent users. It’s best to operate with a capacity margin and use a dynamically elastic server layer, so the system can quickly scale to successfully address traffic surges.
- Data integrity guarantees - for a superior quality of service. As an example, sending the latest odds in the wrong order, or dropping events due to at-most-once properties can shatter users’ trust and see them heading to competitors.
Together, Confluent Cloud and Ably easily provide all of these properties. Thanks to its elasticity, high availability, and reliability, Confluent Cloud is an excellent choice for building interactive betting systems - as showcased at the 2021 Kafka Summit.
Ably is an edge messaging platform designed to distribute and collect low-latency messages to and from web, mobile, and IoT devices, via a fault-tolerant, autoscaling global edge network. Ably is often used as an Apache KafkaⓇ extension for sending mission-critical data to and from end-users at the network edge in realtime.
The newly released Ably Kafka Connector provides seamless no-code integration between Kafka or Confluent and Ably. The Connector enables data processed at sub-second latencies in your Confluent ecosystem to travel to client devices in a fast and dependable way, with Ably as a message broker in the middle. Additionally, Ably makes it easy to stream event data generated by end-users back into your Kafka deployments at any scale with the Firehose service.
Realtime betting solution architecture
To demonstrate how Confluent Cloud and Ably work together when engineering realtime betting functionality, we’ve built a demo with the following architecture:
Whenever new odds are available, they are sent from Confluent Cloud to Ably via the Ably Kafka Connector. Ably then distributes the odds in realtime to any number of Android app users over pub/sub channels.
In the Android app, users can see odds changing in real time, and they can place their bets. Since Ably uses event-driven, bidirectional WebSockets as the primary transport protocol, a user not only receives odds, but they are also able to quickly place their bets over the same connection.
Ably streams the firehose of bets made into Confluent Cloud. ksqlDB is then used to create a materialized view of the current state of the order book, and to push updated odds into the Odds topic. We’ve also built a Python microservice that queries ksqlDB to send personalized notifications to users, making them aware of various events: a win, a loss, or a new race starting soon (more about the microservice later in this article).
From here, the process we’ve covered so far repeats - there’s a continuous loop of data being processed and traveling in real time between the backend and the Android app, with Ably sitting as an edge messaging broker between Kafka and end-users.
System characteristics and benefits
A betting solution built with Confluent Cloud and Ably offers the following advantages:
- Low-latency. Whenever a bet is placed, the information is streamed and processed in milliseconds in the Confluent Cloud topics, while new odds are pushed to end-users via Ably, with sub-second latencies.
- Scalability. Confluent Cloud provides elasticity and self-serve provisioning, allowing you to quickly and effortlessly scale your Kafka ecosystem as needed. Similarly, Ably provides a fully managed layer between the backend and the frontend that can autoscale horizontally to handle millions of concurrently-connected end-user devices.
- Reliable and highly available. Confluent Cloud and Ably are both designed with fault tolerance and high availability in mind. Together, they create a distributed, robust, and trustworthy event-driven pipeline that ensures competitive uptime SLAs, irrespective of infrastructure failures or other faults in the system, such as an availability zone being temporarily unavailable. Confluent Cloud provides a guaranteed 99.95% uptime SLA, and Ably provides a 99.999% uptime SLA, which means the system is always dependably available to end-users.
- Messaging QoS. Confluent Cloud can be configured to ensure ordering and exactly-once semantics, while Ably also guarantees message ordering, delivery, and exactly-once semantics, even in unreliable network conditions. Together, Confluent Cloud and Ably provide data integrity end-to-end: users always receive up-to-date odds, and no bet placed is lost or fails to be processed.
- Ease of use and extensibility. Confluent Cloud and Ably are fully managed solutions. Any system built with these two technologies is low-code and easy to set up and manage, with no infrastructure to maintain. Additionally, Confluent and Ably integrate with many other services (for example, databases and serverless platforms), making it easy to add more components to the architecture and build additional relevant functionality, such as realtime fraud detection.
- Increased engagement and revenue. As low-latency, event-driven solutions, Confluent and Ably enable you to boost engagement by allowing users to bet not only on main events (such as the winner of a race), but even on granular (sub)events that are extremely time-sensitive (like the outcome of a penalty kick). And the more betting opportunities you provide, the higher the chances of increasing your revenue.
Building the realtime betting demo
Before we jump into the step-by-step walkthrough, let’s review the main components we’ll use to build our realtime betting demo:
- Confluent Cloud and ksqlDB, for event streaming and stream processing.
- Ably Kafka Connector for transferring data from Confluent Cloud topics into Ably.
- Ably for distributing odds and push notifications to end-users, and streaming bets they make into Confluent Cloud.
- Android app - for users to view odds changing and place bets in real time.
We’ll now dive into more details, covering how to set up each of these components, and demonstrating how they interact with each other. All of this is presented in the video below.
The rest of this article largely covers the same aspects presented in the video above.
Setting up Confluent Cloud
Setting up the Confluent ecosystem is straightforward and involves a few simple steps. If you don’t have one yet, start by signing up for a Confluent Cloud account.
Once you’ve logged in to Confluent Cloud, create a Kafka cluster (default settings are OK for the purpose of our demo). Note that Confluent gives you the flexibility of selecting the cloud provider and region of your choice to deploy the Kafka cluster. In complementarity, Ably is underpinned by a globally-distributed edge network that guarantees clients are always routed to the Ably datacenter that is closest to them.
Next, provision two Kafka topics, which will be used to store bets made by users, and new odds that are generated whenever bets are placed. By default, each Kafka topic has six partitions; you don’t really need all the six partitions for our demo, but in a real-life scenario, adding more partitions is the primary way to increase Kafka capacity so you can deal with high load.
Now that we have a Kafka cluster and topics up and running, set up a basic ksqlDB application. We’ll use ksqlDB to create a materialized view of incoming bets, and, based on that, generate new odds.
Getting started with Ably
Similar to Confluent, getting the ball rolling with Ably is simple. Start by signing up for an Ably account.
Next, log in to your Ably account dashboard and create an Ably app.
From the Integrations tab of my Ably app, I’ve created a Kafka Rule, which allows us to stream the bets placed in the Android app into the Bets topic in Kafka.
Note that the Kafka Rule is a new feature, and it’s currently in developer preview. It’s also an enterprise-only feature. However, if you plan on building this demo yourself, contact us, and we’ll talk about making the Kafka Rule available to you for testing.
For data distribution to and from the edge, Ably uses channels, which are analogous to Kafka topics. However, Ably channels are provisioned dynamically at the point of use (there’s no need to pre-provision them ahead of time), and they have hierarchical namespaces to enable rule-based processing of messages (e.g., push notifications, firehose rules).
Installing the Ably Kafka Connector
The Ably Kafka Connector is available on Confluent Hub. You can use it with the Confluent Platform, or run it locally with Docker. Installing it is a simple process - follow the GitHub documentation for details.
Initializing the Android app
Grab the code from the GitHub repo and run the project with Android Studio.
Quickly looking at the code, in the following snippet, we initialize the real time connection to Ably:
private void initAbly() throws AblyException {
rtConnection = new AblyRealtime("Your Key Here");
Map<String, String> params = new HashMap<>();
params.put("delta", "vcdiff");
ChannelOptions options = new ChannelOptions;
options.params = params;
Channel channel = rtConnection.channels.get("outbound:odds:event1",options);
channel.subscribe(new Channel.MessageListener() {
We’re also subscribing to the odds channel with in-channel message delta compression enabled. Delta compression helps massively reduce bandwidth and latency for last-mile data distribution by sending only the changes from the previous message to subscribers each time there’s an update, instead of the entire message. This compression is entirely transparent to the user, and the Ably library handles any missing data, and guarantees that the end-user always gets a complete message.
Note that for testing push notifications, you will have to create a Firebase Cloud Messaging account, and set up Ably and Firebase accordingly - read the tutorial for details.
See the solution in action
When you open the Android app, you should see the following:
By clicking on the Activate Device and Deactivate Device buttons, you subscribe and unsubscribe from notifications (which are delivered via Ably’s managed push notifications service). Push notifications may be sent to announce events such as a win, a loss, or a new race starting soon.
Due to push notifications being occasional batch jobs, they are less well-suited to a direct streaming integration between Confluent and Ably. So we’re using a Python microservice that queries ksqlDB via REST, and sends push notifications through Ably. You can find the Python microservice in this demo’s GitHub repo.
Once you’ve clicked the Betting view button, you will be taken to the second screen, where you can see a list of cars with changing odds that you can bet on. To place a bet, just click on any of the car tiles.
Now if you go to the Dev console tab of the Ably app you created earlier, you should see the list of bets placed updating in real time.
Each of these events contains information about when the bet was placed, and the odds at that time. This allows the system to verify that any bet is placed with acceptable odds. We are not demonstrating payment processing here, but in a complete system, this would allow you to accept late bets if the risk isn’t too high.
All bets are steamed from Ably into the Bets topic in Confluent Cloud; this is achieved via the Kafka Rule we mentioned earlier. Bets are then consumed by ksqlDB:
As mentioned before, we use ksqlDB to create a materialized view of the bets placed, and to push updated odds into the Odds topic in Confluent Cloud. With the help of the Ably Kafka Connector, the odds are then sent to Ably, and from Ably to the Android app, via an Ably channel.
This concludes our demo - we hope you’ll find it useful!
Going beyond the demo
So far, we’ve demonstrated the feasibility of using Confluent Cloud and Ably to power realtime betting experiences. But the demo should also serve as inspiration for building scalable, production-ready systems. A realistic high-level architecture for a realtime betting solution built with Confluent Cloud and Ably might look something like this:
Through sink and source connectors, Confluent Cloud makes it easy to add various additional components to your backend, such as databases. Confluent Cloud enables you to effortlessly scale to as many topics as you need, each one storing specific event data (like the list of fixtures), and using Schema Registry to safeguard data formatting.
ksqlDB allows you to create materialized views, and continuously transform, enrich, join together, and aggregate events. You can have various microservices (like the Python one we use in the demo for sending push notifications) querying, consuming, and acting on ksqlDB data.
Ably offers 25+ client SDKs, targeting every major web and mobile platform. This means that almost regardless of what technology you are using on the client-side, Ably can help intermediate the flow of data between the Confluent Cloud backend and end-users. For example, besides Android, you can use Ably to stream data to and from iOS users and web pages.
In addition to powering real-time communication for sports enthusiasts placing bets, Confluent Cloud and Ably can be used to enable capabilities relevant to betting providers, such as risk management, fraud detection, and real time monitoring.
Final thoughts
We hope this blog post helps you understand the benefits of combining Confluent Cloud and Ably when you want to engineer low-latency, scalable, and reliable digital betting experiences for your customers. Together, Confluent and Ably enable you to stream and process high volumes of data in your backend Kafka pipeline, and then broadcast messages in milliseconds to very high and quickly changing numbers of client devices at the network edge, in a fault-tolerant manner, without any update being delivered in the wrong order or more than once.
Although the PoC we’ve covered in this blog post relates to gaming and betting, you can easily combine Confluent Cloud and Ably to deliver scalable realtime digital experiences for a wide variety of use cases and industries, from realtime asset tracking and virtual events to healthcare and EdTech. In any scenario where time-sensitive data needs to be processed and must flow between the data center and client devices at the network edge in (milli)seconds, Confluent Cloud and Ably can help.
Confluent Cloud and Ably are fully managed, serverless solutions, abstracting away the massive infrastructure and engineering overhead and the associated financial costs required to build and manage your own proprietary realtime betting system capable of serving a global user base.
Getting started with Confluent Cloud and Ably is simple - both are almost no-code solutions, and you can be up and running with each in minutes. Additionally, integrating Confluent and Ably is a seamless and quick process made possible by the Ably Kafka Connector - try it out and see for yourself! You might also want to read more about how Ably helps you effortlessly and reliably extend your Kafka pipeline to end-users at the edge.
