Deploying a Multi-Region Kubernetes Cluster image
Webinar

Deploying a Multi-Region Kubernetes Cluster

Published Thursday, Nov 14, 2019

Watch the webinar

Experience Report Revisited: How to Deploy a Kubernetes Cluster in a Multi-Region Application Using CockroachDB

Kubernetes has become the standard for deploying distributed applications. However, orchestrating workloads across regions is still a considerable challenge due to networking, security, and service discovery problems.
In this tech talk, CockroachDB Dedicated engineer Joel Kenny and Kubernetes committer Alex Robinson will detail their real-world experience of running distributed systems across multi-region Kubernetes clusters, including:
  • Common challenges when running Kubernetes across regions
  • Future improvements for multi-region implementations
 

Multi-Region Kubernetes QA

Before the talk from Alex Robinson about how to deploy a multi-region Kubernetes cluster, we answered some questions from the community and offer some examples of how our CockroachDB Dedicated team navigates the challenges of Kubernetes in Multi-Region deployments. Joel Kenny, an engineer from the CockroachDB Dedicated team, addresses the following questions: 

Are virtual networking and secure cross-region communication an option? 

I should note that If you run CockroachDB in “secure mode” we do rely on fully encrypted in-transit communication between all CockroachDB pods in both single and multi-region. If you want to expand this beyond CockroachDB itself to other arbitrary applications your data won’t be encrypted by default. But you could use a service mesh like Istio which will provide you in-transit encryption this also works in single or multi-region Kubernetes clusters.
Regarding region local container registries: AWS, Google Cloud, and Azure provide this feature out of the box. They all support uploading the same container image to different regions just in case you want to host containers in different regions (in case one becomes unavailable).

How does CockroachDB deal with the challenge of orchestrating a Multi-Cloud deployment?

I don’t think there’s great support from the public clouds for multi-cloud deployments (It’s possible that Anthos is useful but we’re still learning about it). For CockroachDB Dedicated we rely on some tools that allow us to orchestrate different cloud providers. One tool we use is Terraform, which allows us to define a common set of resources and then Terraform deals with all the API calls to different cloud providers to spin up different VMs or Kubernetes clusters. We also use a tool called Pulumi, which is like Terraform but it provides better support for interacting with cloud providers using programming languages, such as Python or Nodejs. There’s also an interesting project called Crossplane, which takes on a more Kubernetes-like structure to defining resources in different cloud providers. 

Are multi-region Kubernetes clusters ‘cross-region’ or are they separate clusters?

Kubernetes itself only runs in a single region, and originally, it used to only run in a single availability zone (in a single region), but that’s improved. The focus of the talk in this webinar about how to deploy a multi-region kubernetes cluster is that you can spin up a multi-region application, such as CockroachDB, on two different Kubernetes clusters, (one in each region) and have them all communicate with each other. 
We also have some content about this on our website. There is an example in the GitHub repo and a blog about how to do multi-cloud with Kubernetes. 

How do I access the database publicly by IP (I want to access the database that feeds my applications which are running in different clusters)? 

Within the CockroachDB Dedicated team we create a service type load balancer for each region in a multi-region deployment. So a Kubernetes service will take all of the pods in a cluster and expose them at a single IP address. If this service is a type load balancer then a different cloud provider, such as AWS, Google Cloud, or Azure will create an actual layer for the load balancer and then your external applications can reach CockroachDB via this load balancer. This still means that you have one load balancer per region so if you have a multi-region application my recommendation would be, for each region of your application, to point it to the load balancer that is nearest to the application. 
If you’re running on AWS and you’re using route 53 you could use latency based routing so you’ll have a single URL for all your load balancers and you’ll have a different record set for that same URL and you rely on route 53 to route the DNS request from your application to the load balancer that is nearest.

How close is CockroachDB to full upper-level support for GORM (I would love to deploy multi-region Kubernetes cluster with CockroachDB and GoLang)? 

This is something that we’re currently working on. Check for updates in our community slack channel.  

Speakers

Joel Kenny
Joel Kenny
Software Engineer
Meagan Goldman
Meagan Goldman
Product Marketing Manager
Alex Robinson
Alex Robinson

Additional Resources

Learn how to use CockroachDB
Read the docs
Need help? Get answers on the CockroachDB forum
Ask on the forum
What are you building with CockroachDB?
Share your project