Live betting on ACID: Challenges of building a next-gen gambling app

Imagine you’re watching a baseball game. In the middle of the fifth inning, your pitcher throws a strike, and then another one. The fan sitting next to you jumps to his feet in jubilant celebration.

You could be forgiven for wondering what’s going on. After all, it takes three strikes to get a batter out – why is this guy dancing like the game is already over?

One possible explanation: he was micro betting on that pitch, and he just won a lot of money.

What is micro betting?

Micro betting is a form of fast-paced live betting that allows users to bet on the outcomes of specific events. Micro betting is most commonly seen in sports betting apps, where users can bet on the outcomes of specific in-game events during the game. For example:

• In baseball, a user might bet on the results of an at-bat, or even a single pitch.
• In soccer, a user might bet on the next team to score, or whether a corner kick will be awarded in the next few minutes.
• In American football, a user might bet on the results of the next play, or whether a field goal will be missed.
• In basketball, a user might bet on whether the next basket will be a two- or three-pointer.

Of course, these are just examples. Micro betting options vary widely across different betting platforms and different sports.

What they have in common is the user experience. Generally, microbetters are users who are actively watching the game in question, with their betting app open and ready. When they want to bet on the outcome of an in-game event, they can quickly check the odds, place their bet, and then see the outcome in a matter of minutes (in the case of betting on something like the next goal in a soccer game) or even seconds (in the case of betting on something like a single pitch in baseball or a field goal kick in American football).

Micro betting is just one type of live betting – placing bets on games while they’re in progress. It’s also possible to place other types of live bets. For example, a user may bet on which team’s going to win a game while the game is already in progress.

Offering live betting options has some clear advantages for gambling apps, as giving users the opportunity to place many small bets over the course of a game helps keep them engaged with the app, and may lead to a higher overall number of bets placed per game. The gambling industry is projected to grow massively over the next few years, and is becoming increasingly competitive. One of the major reasons for that is growth in revenue that’s driven by an increasing number of highly-engaged live betters.

The technical challenges of live betting

Live betting is highly lucrative, and increasingly expected by users, particularly in the context of sports betting apps. But actually implementing it presents some significant technical challenges. Some of these challenges will be specific to each application’s unique architecture and tech stack, but many of them relate to how applications work with their databases.

So what are the database-related challenges involved in implementing live betting?

Minimizing latency. Ensuring that users have a consistent, lag-free experience regardless of their location and internet connection means minimizing bandwidth usage and latency wherever possible. On the developer end, this means architecting the application and database to ensure transactions aren’t sending any superfluous information, and probably also using a multi-region database to place relevant data on a node that is somewhere geographically close to the user.

Scaling to handle sudden loads. While interest in a huge sporting event like the Champions League final is quite predictable, live betting trends are more difficult to predict, since interest in the outcome of one event might increase or decrease based on what’s happening before that event in the game. Moments of unexpected drama can cause a sudden flurry of betting, and the database needs to be able to handle that without getting bogged down.

Avoiding downtime (and data loss). While there’s really no industry where users consider downtime or data loss acceptable, outages can be especially damaging in the context of micro betting, where even a minute or two of downtime can prevent users from placing bets they wanted to make. Extremely high availability is essential, which means working with a distributed database system is necessary.

Ensuring transaction order correctness. Users should only win bets about events if they’ve placed them before the event occurs, and their winnings or losses should be based on the odds they were given at the time they placed the bet. That sounds obvious, but in the context of live betting it can be difficult to accomplish, as there may only be seconds between the time when a user’s bet is placed and the in-game event occurs, and the odds are changing constantly. The database must be able to determine a correct sequence of transactions/events, which can become difficult when dealing with distributed systems and high transaction volumes.

Ensuring data is consistent and correct. Gambling applications are dealing with huge amounts of money, so there’s really no room for inconsistency or errors in the data.

Of course, there are other technical challenges, too. Operating a gambling service at scale requires operating many data-based services, including things such as fraud detection, logging, regulatory control services, etc. A large gambling app may have hundreds of data-based services all operating in concert, with some reliant on streaming data and other less time-sensitive services using a batch-oriented approach. In the context of this article, we cannot possibly cover all of that complexity, but the examples above provide a good idea of what developers are facing when they’re trying to architect their application and its databases to function at scale in the world of live betting.

Many of these challenges can be addressed by the database itself, so let’s take a look at some of the available database options and their advantages and disadvantages in the context of addressing the challenges inherent in operating a micro betting service.

SQL vs. NoSQL vs. Distributed SQL

Traditional SQL

The relational database is a tried-and-tested classic that businesses have been using for decades. While it comes in many flavors – Postgres, MySQL, Oracle, etc. – for the most part they share the same strengths and weaknesses.

In the context of gambling applications, the primary strength of SQL databases is that they can offer strong data accuracy and integrity because they support ACID transactions. (At least in theory – in practice many relational databases don’t provide ACID guarantees by default.

ACID guarantees are helpful for preserving business rule invariants and ensuring correct execution in live betting applications by ensuring that transactions have:

• Atomicity – each transaction is treated as a single, all-or-nothing unit and cannot be partially executed.
• Consistency – each transaction can only commit a change if it meets the database’s rules for validity and referential integrity.
• Isolation – concurrently-processed transactions will result in the same final database state as if they had been processed sequentially.
• Durability – each transaction, once committed, will remain committed even in the case of node or server failure, etc.

While business processes that span multiple online betting services leverage event-driven orchestration and choreography to implement transactions, there still needs to be a single source of truth per domain aggregate with local ACID transactions preserving strong invariants. ACID transactions therefore provide an important mechanism even in a strongly event-driven domain.

Traditional SQL databases also offer developers other advantages such as familiarity – most developers already know SQL – but they come with significant downsides, too. Most of these technologies are old – in some cases, decades old – and they weren’t built for distributed computing or for serving global markets.

Getting a traditional SQL database to operate at the scale (both in terms of user volume and geography) required for a modern gambling application would be a significant challenge requiring a lot of manual work, not to mention the ongoing ops burden that would be required to maintain it.

NoSQL

The emergence of NoSQL databases gave gambling app developers a more modern option that provided elastic scale, and in some cases global distribution, without all of the manual lifting and retrofitting that would be required to get the same thing out of a traditional SQL database.

NoSQL databases, however, don’t offer the strong data accuracy and consistency guarantees found in traditional SQL databases. This has required developers of gambling applications to make some compromises when it comes to consistency, and to embrace some limitations in their database design.

Gambling service providers are (understandably) typically pretty reluctant to talk about their tech infrastructure, but we can see an example of some of the kinds of compromises NoSQL requires by looking at how gambling company Bet365 approached its initial transition from SQL to NoSQL.

Having pushed its existing SQL architecture to its limits, Bet365 wanted to switch to an easier-to-scale NoSQL database. But the need for data consistency limited its options. And to achieve eventual consistency with the system they ultimately chose, they also had to use CRDT datatypes.

CRDTs make achieving consistency easier in distributed systems by offering a way that multiple replicas of the same data can be updated at the same time without centralized coordination. However, they can be tremendously limiting, and it will require most developers a good amount of time and effort to wrap their heads around how to work with these limitations and to build out and test each potential solution to ensure eventual consistency can be achieved.

In short, NoSQL databases offer easier scalign than a traditional relational database, but it can be quite challenging to get them to meet the consistency needs of modern gambling applications. While it’s possible to work around the lack of consistency inherent in many NoSQL systems, it can take a lot of time and effort, and often involves embracing major restrictions such as CRDTs.

Distributed SQL

The newest option on the block, distributed SQL databases marry the familiarity and consistency offered by traditional relational databases with the elastic scalability of NoSQL databases. Because they’re distributed, they typically can offer very high availability, too.

Not all distributed SQL databases are the same, of course, but let’s take a look how one such database, CockroachDB, meets all five of the original database requirements for live betting that we listed, with no need for compromise:

Minimizing latency. CockroachDB is a distributed database that supports multi-region and data geolocation down to the row level. This allows gambling companies to store data close to the users who need it, which is useful for minimizing latency and often also useful for regulatory compliance.

Scaling to handle sudden loads. CockroachDB was built to scale elastically, and it’s already used in production by companies such as Comcast and Netflix that operate at tremendous scale.

Avoiding downtime (and data loss). CockroachDB’s distributed nature makes it tremendously resilient. And because it’s not affiliated with a cloud vendor (unlike distributed SQL databases Amazon Aurora and Google Spanner), it can even survive cloud provider outages if you opt for a multi-cloud or hybrid cloud/self-hosted deployment.

Ensuring transaction order correctness. CockroachDB uses the highest SQL isolation level, serializable isolation. This ensures that transactions are processed in a manner consistent with serial execution, even when dealing with large numbers of concurrent transactions.

Ensuring data is consistent and correct. CockroachDB offers the ironclad correctness and consistency guarantees – even stronger than many traditional relational databases provide by default, as it supports truly ACID transactions with serializable isolation. This means that developers can achieve consistency without having to use restrictive datatypes such as CRDTs.

Next-gen workloads, meet a next-gen database

Every use case is slightly different, and we won’t pretend there’s any database out there that’s perfect for every single gambling app. However, the rise of the distributed SQL database certainly seems to have come at a perfect time for gambling app developers. Without the need to find a lesser-of-two-evils compromise, developers can stop having to worry about their databases and focus on building the next generation of live betting applications.