The ultimate battle for consistency: Concurrent betting at massive scale

The odds are in your favor with CockroachDB

Once they were ready to take advantage of a managed service offering which includes additional features like change data capture (CDC) and the ability to partition data by location, the team upgraded to CockroachDB Dedicated. With this plan, they can start small and scale really fast as they onboard new customers. Dan and his team remarked how impressed they have been with the guidance and support from the Cockroach Labs team.

“The team at Cockroach Labs treats us as if we were at Amazon and giving them $1 million bucks a month. We talk to the people that help us solve things, move forward, have suggestions about what we should do. It’s proven to be extremely valuable and allows us to focus on building new features.” - Dan Goodman, CTO and Co-Founder

“CockroachDB is being used as a system of record for all of their data which includes payments, (bets, account values, etc), pools, and gamification features. They also use TimescaleDB for managing their time-series data and Redis for processing real-time odds and values. Ultimately all of their data makes its way into CockroachDB which serves as their single source of truth.

They are building on distributed microservices-based systems which gives them a lot of flexibility when it comes to deployment. At the moment, they are on a single cluster with 2 core nodes in one AWS region. However, they are designing their infrastructure with the notion that they will expand into multiple regions and into Europe as the business grows. This is when setting locality to domicile data close to users will become extremely important in order to comply with GDPR and additional government regulatory requirements specifically for the gambling industry. Each country within the EU has its own rules for how they organize their gambling services which makes data locality essential. 

Their developers are also using the free tier of CockroachDB Serverless for testing and development. They set up a local dev environment that’s connected to a serverless cluster. This allows their developers to go through quick dev cycles, push out new features, collaborate on the same data, and reduce iteration time – at zero cost and zero liability of breaking anything. 

Here’s how it works: a user enters a bet into the system which asks CockroachDB if the pool is open (yes/no). If the pool is open, the bet is inserted and asks CockroachDB if there’s enough money in the user’s account to place the bet (yes/no). If there is enough money, the value is detracted from the user’s account and the bet is successfully placed.”

“The bet becomes an event which travels downstream in the form of an event source pipeline. (Also known as CDC which is included with CockroachDB – many other systems require you to manually build an event source pipeline). Once confirmed, the betting API talks to Redis to update the real time value of the entire pool and new odds. It also talks to TimescaleDB and fetches the time-series data which is stored as a record in CockroachDB. 

For this system to work, CockroachDB’s CDC capabilities are extremely important since it creates an audit log of every transaction (bet slip, pool, user account value) and constantly updates it to Amazon S3. This is crucial for two reasons 1) they can replay every transaction state in the event that they need to debug the system or trace money 2) if regulators ever need to see a report for fraud investigation, they can go back and show every transaction.”

“We keep everything downstream from CockroachDB which is our single source of truth. It’s the most resilient, distributed, and highly available system. Everything else comes as a sort of `quality of life` feature that we get downstream. Our time-series doesn't have to work for our system to behave properly. Our Redis can vanish and operationally we would still be OK because we would just fall back to CockroachDB.” - Dan Goodman, CTO and Co-Founder

Bet on a distributed environment and know your data

“Dan’s best piece of advice for people who are new to distributed systems: understand your data and how it interacts with the system your data lives on. For Ultimate Tournament, they knew they would be primarily dealing with two types of data: real-time transactional data and time-series data. 

Both of these data types can create high cardinality which refers to columns with values that are very uncommon or unique. For example, any time a bet is placed for a particular pool it has an associated USER_ID and SLIP_ID which is unique to that specific bet slip. Now imagine 100 different people with unique USER_ID and SLIP_ID are placing bets on that same pool at the same time. This data set starts to grow exponentially and the more unique values it has, the higher the cardinality. If you are not using the right solution and properly organizing your data, high cardinality data can damage insert rates and query performance.

Ultimate Tournament’s transactional data includes payments, bets, pools, user IDs, etc. that needs to be consistently updated (placed, canceled, processed) in real-time. Right now, they provide betting on 5,000 matches a month. This cardinality is low; a relational database can easily handle that volume. However, when they open up their platform to casinos with millions of users, it presents the challenge of accommodating a large influx of concurrent transactions at once, in real-time. They also need to offer the ability to manipulate bets (place, cancel, modify) as many times as the users want, and of course all this data has to be accurate. Most databases simply can’t handle this complexity which is why they use CockroachDB for transactional workloads. 

They also have a lot of time-series data which can become high cardinality data since it tracks a series of data points collected over time intervals (i.e. it creates a large number of unique values in a set). That’s why Ultimate Tournament is using TimescaleDB (a relational database with support for time-series) to help optimize their time-series workloads which is important to keep cardinality low. 

In order to work with their data, Dan created two development rules: 

1. no full table scans
   
2. no hot keys and hot rows (rows that are frequently added, deleted, modified, or queried)
   
   Dan needed to make sure they were organizing data in CockroachDB the right way to reduce the risk of high cardinality data slowing the system. At the same time, he wanted to optimize performance. They designed their data so they can read it efficiently which meant intelligently designing primary keys upfront so they can get multiple access patterns out of the same primary key.  
   
   This also meant designing indexesfor specific columns, which creates a copy of the columns and then sorts their values (without sorting the values in the table itself). After a column is indexed, they can easily filter its values using the index instead of scanning each row one-by-one. On large tables, this greatly reduces the number of rows scanned (i.e. no full table scans) which enables them to execute queries exponentially faster. 
   
   Now, they can go one step further and create multiple indexes of the same table and pin the indexes to different regions. This feature helps avoid performance bottlenecks. For example, say they are constantly going back to the same pool to make sure it’s still open; this can get really “hot”. Instead, they have multiple indexes of the same table in different regions. So they aren’t jumping to access a certain node, they can access any node because there is consistent copy of that data on every node (thanks to ranges which are part of CockroachDB’s distribution layer). This eliminates the hotkey problem. 
   
   As you can tell, the data that’s generated from gambling applications is complex.”

What’s Next

“The team describes their app as the “Robinhood of eSports betting” since it allows eSports betting to be intuitive for everyone, and their platform as the “Shopify of eSports betting” since companies can white label the platform and make it their own. They have two main focuses right now which include getting more gamers to join their community and piloting eSportsbook™️ to a select group of testers. 

The Ultimate Tournament Community is already 10,000+ members strong. Gamers can sign up to be part of the community and enter to play in weekly tournaments that have prizes ranging from $100-$2,000. 

The early access to eSportsbook™️ lets users register, customize accounts, and place fake money bets on real matches. These testers are providing Ultimate Tournament with the necessary KPIs, feedback, and stress tests needed to squash bugs and optimize their platform. 

In early 2022, they plan to open eSportsbook™️ up to the public. They hope to have several operators white labeling their platform within the next year. Along the way, they will be relying on CockroachDB to grow with them as their business scales. Dan and his team feel confident that they will be able to build “the best eSports betting platform on the face of the earth”. 

Let the games begin.”

“We only have one really good shot at this because the [gambling] space is extremely difficult for startups. Getting it right up front is not just a question of end user experience, but a question of the business surviving. Making sure that we have consistency guarantees as well as high availability up front were non-negotiables. When you start lining up the requirement boxes next to the number of databases that can check them, it starts shrinking fast. For us, all the arrows were pointing to CockroachDB.” - Dan Goodman, CTO and Co-Founder