Egress Perimeter Controls for CockroachDB Advanced

On this page

This page describes how Egress Perimeter Controls can enhance the security of CockroachDB Advanced clusters, and gives an overview of how to manage a cluster's egress rules.

Why use Egress Perimeter Controls

CockroachDB Advanced clusters access external resources for many purposes:

• Managing backups as part of a disaster recovery plan
• Using Change data capture (CDC) changefeeds
• Exporting data
• Exporting logs

By default, clusters can access external resources via the internet without restriction, and even private clusters can access their private network. This potentially leaves a cluster open to a data exfiltration scenario, wherein an attacker, often a malicious insider, steals data by sending backups, changefeeds, data, or logs to a source that they control.

Operators of CockroachDB Advanced clusters can mitigate against this risk by using Egress Perimeter Controls, which enable Cluster Administrators to restrict egress to a list of specified external destinations. This adds a strong layer of protection against malicious or accidental data exfiltration. Along with other measures such as Private Clusters, Egress Perimeter Controls are an important component in an overall strategy for maximizing network security.

Further reading: review how CockroachDB products differs in advanced security features.

Before you begin

• Egress Perimeter Controls are supported on AWS and GCP for the following deployment types:

  • CockroachDB Advanced clusters with enhanced security features.
  • CockroachDB Advanced Private Clusters.

  Egress Perimeter Controls are not supported for CockroachDB Advanced on Azure.

• You need a service account with the Cluster Administrator role on clusters in your organization. You can provision service accounts and API keys in CockroachDB Cloud Console. Refer to Service Accounts.

Initialize your shell with your API key and Cluster id

1. Inspect your cluster in the clusters page in the CockroachDB Cloud console.

2. Find your cluster's universally unique identifier (UUID). To do this, select your cluster from the clusters page in the console. The UUID will appear in the URL of the overview page for that specific cluster, in the format:

   https://cockroachlabs.cloud/cluster/{ your cluster's UUID }/overview

3. Save your API key and cluster UUID as environment variables.

   CC_API_KEY={ your API key }
   CLUSTER_ID={ your cluster UUID }
   

4. Inspect your cluster with the Cloud API:

   curl --request GET \
   --header "Authorization: Bearer $CC_API_KEY" \
   --url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID"
   

   
   {
     "id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
     "name": "docstest",
     "cockroach_version": "latest-v22.2-build(sha256:e42c4de8577556132120a9ab07efc1a2a96779c028ebab99223d862d9792428b)",
     "plan": "ADVANCED",
     "cloud_provider": "AWS",
     "account_id": "1234567890",
     "state": "CREATED",
     "creator_id": "b7d7bedc-b8f3-4a98-bd2c-4ba2bf9adbe1",
     "operation_status": "CLUSTER_STATUS_UNSPECIFIED",
     "config": {
       "dedicated": {
         "machine_type": "m5.large",
         "num_virtual_cpus": 4,
         "storage_gib": 15,
         "memory_gib": 8,
         "disk_iops": 225
       }
     },
     "regions": [
       {
         "name": "us-west-2",
         "sql_dns": "docstest-6qsh.aws-us-west-2.crdb.io",
         "ui_dns": "admin-docstest-6qsh.aws-us-west-2.crdb.io",
         "internal_dns": "",
         "node_count": 3
       }
     ],
     "created_at": "2022-10-27T18:03:47.862079Z",
     "updated_at": "2022-10-27T18:24:58.111198Z",
     "deleted_at": null
   }
   

Use a deny-by-default egress traffic policy

1. Make a POST request ordering the API to update your cluster's egress policy from default allow-all to deny-all. This allows you to add egress rules to explicitly allow egress traffic from the cluster to certain external destinations.

   curl --request POST \
   --header "Authorization: Bearer $CC_API_KEY" \
   --header 'Cc-Version: latest' \
   --url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/egress-traffic-policy" \
   --data '{"allow_all":false}'
   

   {}
   

Create an egress rule to allow a destination

An egress rule is created with the following attributes:

• name: A name for the rule.
• type: Whether the destination will be specified as a fully-qualified domain name (FQDN), or as a range of IP addresses specified in CIDR notation. Value is "FQDN" or "CIDR".
• destination: Either a fully qualified domain name, for example www.cockroachlabs.com, or a CIDR range, for example, 123.45.67.890/32.
• ports: An array of allowed ports, for example [44,8080].
  Warning:

  By default, all ports are allowed.
• paths: Deprecated. This field is ignored and will be removed in the future. An egress rule applies to all paths.
• description: The intended purpose of egress to the destination.

The following steps create one FQDN rule and one CIDR rule.

1. First, create a YAML manifest for a FQDN-based rule. This example allows egress traffic from the cluster to storage.googleapis.com (the Google Cloud Platform (GCP) storage API) on ports 80 and 443.

   ---
   name: "roach-buckets"
   type: "FQDN"
   destination: "storage.googleapis.com"
   ports: [80, 443]
   description: 'egress for GCP storage API'
   

2. Next, create a YAML manifest for a CIDR-based rule. This example adds egress traffic from the cluster to the IPv4 subnet 123.34.62.123/32 (which matches a single IPv4 address) on port 443, which may be, for example, the load balancer for a Kafka cluster running in your organization's internal network or private cloud.

   --- # egress-rule2.yml
   name: "roach-kafka"
   type: "CIDR"
   destination: "123.34.62.123/32" # replace with Kafka cluster CIDR range
   ports: [443]
   description: 'egress for Kafka'
   

3. Use Ruby (or another technique), to compile human-editable YAML into API-parsable JSON:

   ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' < egress-rule1.yml > egress-rule1.json
   ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' < egress-rule2.yml > egress-rule2.json
   

4. Use the shell utility jq to inspect the JSON payload:

   Note:

   On macOS, you can install jq from Homebrew: brew install jq

   cat egress-rule1.json |jq
   cat egress-rule2.json |jq
   

   {
     "name": "roach-buckets",
     "type": "FQDN",
     "destination": "storage.googleapis.com",
     "ports": [
       443,
       80
     ],
     "description": "egress for GCP storage API"
   }
   {
     "name": "roach-kafka",
     "type": "CIDR",
     "destination": "123.34.62.123/32",
     "ports": [
       443
     ],
     "description": "egress for Kafka"
   }
   

5. Make a POST request to create each rule from its JSON payload.

   curl --request POST \
   --header "Authorization: Bearer $CC_API_KEY" \
   --url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules" \
   --data "@egress-rule1.json"
   
   curl --request POST \
   --header "Authorization: Bearer $CC_API_KEY" \
   --url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules" \
   --data "@egress-rule2.json"
   

   {
     "Rule": {
       "id": "564a25da-b99c-4e08-9ec3-483c0f1bc620",
       "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
       "name": "roach-buckets",
       "type": "FQDN",
       "state": "PENDING_CREATION",
       "crl_managed": false,
       "destination": "storage.googleapis.com",
       "ports": [
         443,
         80
       ],
       "description": "egress for GCP storage API",
       "created_at": "2022-10-27T19:35:51.435571Z"
     }
   }
   {
     "Rule": {
       "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
       "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
       "name": "roach-kafka",
       "type": "CIDR",
       "state": "PENDING_CREATION",
       "crl_managed": false,
       "destination": "123.34.62.123/32",
       "ports": [
         443
       ],
       "description": "egress for Kafka",
       "created_at": "2022-10-27T19:36:25.670162Z"
     }
   }
   

   Warning:

   Your cluster's firewall behavior is enforced asynchronously after the API response. After submitting the request, check your egress rules to confirm that the new rules have been created.

Check the status of a rule

curl --request GET \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/$RULE_ID"

{
  "rule": {
    "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-kafka",
    "type": "CIDR",
    "state": "ACTIVE",
    "crl_managed": false,
    "destination": "123.34.62.123/32",
    "ports": [
      443
    ],
    "description": "egress for Kafka",
    "created_at": "2022-10-27T19:36:25.670162Z"
  }

Check egress rules for a cluster

curl --request GET \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules"

{
  "rules": [
    {
      "id": "564a25da-b99c-4e08-9ec3-483c0f1bc620",
      "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
      "name": "roach-buckets",
      "type": "FQDN",
      "state": "ACTIVE",
      "crl_managed": false,
      "destination": "storage.googleapis.com",
      "ports": [
        443,
        80
      ],
      "description": "egress for GCP storage API",
      "created_at": "2022-10-27T19:35:51.435571Z"
    },
    {
      "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
      "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
      "name": "roach-kafka",
      "type": "CIDR",
      "state": "ACTIVE",
      "crl_managed": false,
      "destination": "123.34.62.123/32",
      "ports": [
        443
      ],
      "description": "egress for Kafka",
      "created_at": "2022-10-27T19:36:25.670162Z"
    }
  ],
  "pagination": null
}

Remove a rule

To delete a rule, make DELETE request to the rule's path.

curl --request DELETE \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://cockroachlabs.cloud/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/$RULE_ID"

{
  "Rule": {
    "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-kafka",
    "type": "CIDR",
    "state": "PENDING_DELETION",
    "crl_managed": false,
    "destination": "123.34.62.123/32",
    "ports": [
      443
    ],
    "description": "egress for Kafka",
    "created_at": "2022-10-27T19:36:25.670162Z"
  }
}

Rule statuses

The API displays the following statuses for a rule, when you [inspect an egress rule] or [list a cluster's egress rules]:

• ACTIVE: The rule has been successfully enforced in your cluster's network firewall and egress is currently allowed to the specified destination.
• PENDING_CREATION: Implementation of a new rule is in process. The behavior of the cluster's network firewall may not yet reflect the new rule, so egress may not yet be allowed to the selected destination.
• CREATION_FAILED: Something went wrong in the process of implementing a rule. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.
• PENDING_UPDATE: An PATCH request to update a rule is in process. The behavior may not yet be enforced in your cluster's firewall.
• PENDING_DELETION: The rule-removal is being enforced. The behavior of the cluster's network firewall may still reflect the target rule, so egress may still be allowed to the destination.
• DELETION_FAILED: A rule update or a rule removal has failed. The behavior of the cluster's network firewall may still reflect the previous state of the rule. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.
• INCONSISTENT: This indicates that a rule cannot apply consistently across all of the regions or compute resources to which it is meant to apply. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.

Was this helpful?