Export Metrics From a CockroachDB Advanced Cluster

Exporting metrics to Amazon CloudWatch is only available on CockroachDB Advanced clusters that are hosted on AWS. If your CockroachDB Advanced cluster is hosted on Azure, you can export metrics to Azure Monitor. If your CockroachDB Advanced cluster is hosted on GCP, you can export metrics to Datadog or Prometheus instead.

Perform the following steps to enable metrics export from your CockroachDB Advanced cluster to Amazon CloudWatch.

1. Create the desired target Amazon CloudWatch log group by following the Create a log group in CloudWatch instructions. If you already have a log group created, you may skip this step. When your CockroachDB Advanced cluster emits metrics to Amazon CloudWatch, they are written to this log group.

2. Find your CockroachDB Advanced cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. The ID should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.

3. Determine your cluster's cloud provider account ID. This command uses the third-party JSON parsing tool jq to isolate just the needed account_id field:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id} \
     --header 'Authorization: Bearer {secret_key}' | jq .account_id
   

   Where:

   • {your_cluster_id} is the cluster ID of your CockroachDB Advanced cluster as determined in step 2.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for more details.

4. Create a cross-account IAM role in your AWS account:

   1. In the AWS console, visit the IAM page.
   2. Select Roles and click Create role.
   3. For Trusted entity type, select AWS account.
   4. Choose Another AWS account.
   5. For Account ID, provide the CockroachDB Advanced cloud provider account ID from step 3.
   6. Finish creating the IAM role with a suitable name. These instructions will use the role name CockroachCloudMetricsExportRole. You do not need to add any permissions.
   Note:

   You will need the Amazon Resource Name (ARN) for your cross-account IAM role later in this procedure.

5. Select the new role, and create a new policy for this role. These instructions will use the policy name CockroachCloudMetricsExportPolicy.

6. Select the new policy, and paste the following into the Permissions tab, with the {} JSON option selected:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "logs:CreateLogGroup",
                   "logs:CreateLogStream",
                   "logs:DescribeLogGroups",
                   "logs:DescribeLogStreams",
                   "logs:PutRetentionPolicy",
                   "logs:PutLogEvents"
               ],
               "Effect": "Allow",
               "Resource": [
                   "arn:aws:logs:*:{your_aws_acct_id}:log-group:{log_group_name}",
                   "arn:aws:logs:*:{your_aws_acct_id}:log-group:{log_group_name}:log-stream:*"
               ]
           }
       ]
   }
   

   Where:

   • {your_aws_acct_id} is the AWS Account ID of the AWS account where you created the CockroachCloudMetricsExportRole role, not the cloud provider account ID of your CockroachDB Advanced cluster from step 3. You can find your AWS Account ID on the AWS IAM page.
   • {log_group_name} is the target Amazon CloudWatch log group you created in step 1.

   This defines the set of permissions that the CockroachDB Advanced metrics export feature requires to be able to write metrics to CloudWatch.

7. Copy the Amazon Resource Name (ARN) of the CockroachCloudMetricsExportRole role found under Summary, which is needed for the next step.

8. Issue the following Cloud API command to enable metrics export for your CockroachDB Advanced cluster:

   curl --request POST \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/cloudwatch \
     --header "Authorization: Bearer {secret_key}" \
     --data '{"target_region": "{aws_region}", "role_arn": "arn:aws:iam::{role_arn}:role/CockroachCloudMetricsExportRole", "log_group_name": "{log_group_name}"}'
   

   Where:

   • {cluster_id} is your CockroachDB Advanced cluster ID as determined in step 2.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.
   • {aws_region} is your AWS region, like us-east-1.
   • {role_arn} is the ARN for the CockroachCloudMetricsExportRole role you copied in step 7. If you used a different role name there, be sure to use your role name in place of CockroachCloudMetricsExportRole in the above command.
   • {log_group_name} is the target Amazon CloudWatch log group you created in step 1. This must be the same group name you provided in step 6.

   Specifying an AWS region (to {aws_region}) that you do not have a cluster in, or a region that only partially covers your cluster's nodes will result in missing metrics.

9. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/cloudwatch \
     --header "Authorization: Bearer {secret_key}"
   

   Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and metrics will begin appearing in CloudWatch under the log group you created in step 1. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some metrics appear even before the GET command returns an ENABLED status.

10. Once metrics export has been enabled, you can access metrics from your CockroachDB Advanced cluster directly in Amazon CloudWatch.

To enable metrics export for your CockroachDB Advanced cluster to Datadog, you can either:

• Use the Cloud Console, following the Monitor CockroachDB Cloud with Datadog instructions.

OR

• Use the Cloud API, following the steps below.

1. Find your CockroachDB Advanced cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. The ID should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.

2. Determine the Datadog API key you'd like to use. If you don't already have one, follow the steps to add a new Datadog API key.

3. Issue the following Cloud API command to enable metrics export for your CockroachDB Advanced cluster:

   curl --request POST \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/datadog \
     --header "Authorization: Bearer {secret_key}" \
     --data '{"site": "{datadog_site}", "api_key": "{datadog_api_key}"}'
   

   Where:

   • {cluster_id} is your CockroachDB Advanced cluster ID as determined in step 1, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.
   • {datadog_site} is your Datadog site. Valid sites are: US1, US3, US5, US1_GOV, and EU1.
   • {datadog_api_key} is the Datadog API key determined in step 2.

4. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/datadog \
     --header "Authorization: Bearer {secret_key}"
   

   Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and metrics will begin appearing in the Datadog interface. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some metrics appear even before the GET command returns an ENABLED status.

5. Once metrics export has been enabled, you can access metrics from your CockroachDB Advanced cluster directly in Datadog's Metrics Explorer, or via Datadog's notebook or dashboard features.

6. Review enable percentiles for selected metrics. Configure metrics as necessary.

1. Find your CockroachDB Advanced cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. The ID should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.

2. Issue the following Cloud API command to enable metrics export for your CockroachDB Advanced cluster:

   curl --request POST \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/prometheus \
     --header "Authorization: Bearer {secret_key}"
   

   Where:

   • {cluster_id} is your CockroachDB Advanced cluster ID as determined in step 1, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

3. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/prometheus \
     --header "Authorization: Bearer {secret_key}"
   

   Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete. The response also includes targets, a map of scrape endpoints exposing metrics to regions. For example:

   {
     "cluster_id": "f78b7feb-b6cf-4396-9d7f-494982d7d81e",
     "user_message": "This integration is active.",
     "status": "ENABLED",
     "targets": {
       "us-east4": "https://cockroachlabs.cloud/api/v1/clusters/f78b7feb-b6cf-4396-9d7f-494982d7d81e/metricexport/prometheus/us-east4/scrape"
     }
   }
   

   There is a separate scrape endpoint per region if you have a multi-region cluster.

4. You can test a scrape endpoint by using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/prometheus/{cluster_region}/scrape \
     --header "Authorization: Bearer {secret_key}"
   

   Where:

   • {cluster_id} is your CockroachDB Advanced cluster ID as determined in step 1, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
   • {cluster_region} is a region of your CockroachDB Advanced cluster as shown in the targets of step 3, such as us-east4. You can also find your cluster’s region(s) on the Cluster Overview page.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

   Metrics are labeled with the cluster name and ID, node, organization name, and region. The beginning lines of a metrics scrape response follows:

   # HELP crdb_cloud_addsstable_applications Number of SSTable ingestions applied (i.e. applied by Replicas)
   # TYPE crdb_cloud_addsstable_applications counter
   crdb_cloud_addsstable_applications{cluster="test-gcp",instance="172.28.0.167:8080",node="cockroachdb-j5t6j",node_id="1",organization="CRL - Test",region="us-east4",store="1"} 0
   crdb_cloud_addsstable_applications{cluster="test-gcp",instance="172.28.0.49:8080",node="cockroachdb-ttzj8",node_id="3",organization="CRL - Test",region="us-east4",store="3"} 0
   crdb_cloud_addsstable_applications{cluster="test-gcp",instance="172.28.0.99:8080",node="cockroachdb-r5rns",node_id="2",organization="CRL - Test",region="us-east4",store="2"} 0
   # HELP crdb_cloud_addsstable_copies number of SSTable ingestions that required copying files during application
   # TYPE crdb_cloud_addsstable_copies counter
   crdb_cloud_addsstable_copies{cluster="test-gcp",instance="172.28.0.167:8080",node="cockroachdb-j5t6j",node_id="1",organization="CRL - Test",region="us-east4",store="1"} 0
   crdb_cloud_addsstable_copies{cluster="test-gcp",instance="172.28.0.49:8080",node="cockroachdb-ttzj8",node_id="3",organization="CRL - Test",region="us-east4",store="3"} 0
   crdb_cloud_addsstable_copies{cluster="test-gcp",instance="172.28.0.99:8080",node="cockroachdb-r5rns",node_id="2",organization="CRL - Test",region="us-east4",store="2"} 0
   # HELP crdb_cloud_addsstable_proposals Number of SSTable ingestions proposed (i.e. sent to Raft by lease holders)
   # TYPE crdb_cloud_addsstable_proposals counter
   crdb_cloud_addsstable_proposals{cluster="test-gcp",instance="172.28.0.167:8080",node="cockroachdb-j5t6j",node_id="1",organization="CRL - Test",region="us-east4",store="1"} 0
   crdb_cloud_addsstable_proposals{cluster="test-gcp",instance="172.28.0.49:8080",node="cockroachdb-ttzj8",node_id="3",organization="CRL - Test",region="us-east4",store="3"} 0
   crdb_cloud_addsstable_proposals{cluster="test-gcp",instance="172.28.0.99:8080",node="cockroachdb-r5rns",node_id="2",organization="CRL - Test",region="us-east4",store="2"} 0
   

5. Once metrics export has been enabled and the scrape endpoint(s) tested, you need to configure your metrics collector to periodically poll the scrape endpoint(s). Configure your Prometheus configuration file's scrape_configs section as in the following example:

   global:
     scrape_interval: 10s
     evaluation_interval: 10s
   
   # Prometheus configuration for CockroachDB Advanced for a single region
   scrape_configs:
     - job_name: '{job_name}'
       metrics_path: '/api/v1/clusters/{cluster_id}/metricexport/prometheus/{cluster_region}/scrape'
       static_configs:
         - targets: ['cockroachlabs.cloud']
       scheme: 'https'
       authorization:
         credentials: '{secret_key}'
   

   Where:

   • {job_name} is a job name you assign to scraped metrics by default, such as scrape-cockroach-us-east4.
   • {cluster_id} is your CockroachDB Advanced cluster ID as determined in step 1, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
   • {cluster_region} is a region of your CockroachDB Advanced cluster as shown in the targets of step 3, such as us-east4. You can also find your cluster’s region(s) on the Cluster Overview page.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

Exporting metrics to Azure Monitor is available only on CockroachDB Advanced clusters that are hosted on Azure. If your CockroachDB Advanced cluster is hosted on AWS, you can export metrics to Amazon CloudWatch. If it is hosted on GCP, you can export metrics to Datadog or Prometheus instead.

To enable metrics export to Azure Monitor:

1. Find your CockroachDB Advanced cluster ID, which will be used in step 5:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. The ID should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.

2. Create a private key and certificate pair, using a tool such as OpenSSL. Refer to Create the CA key and certificate pair. In this example, the private key file is named ca.key and the certificate file is named ca.crt. If you use other file names, adjust the commands accordingly. The certificate file will be uploaded in step 3.

   1. Concatenate the certificate and key files using the following command. The string in the concatenated file will be used in step 5.

   cat ca.crt ca.key | awk '{printf "%s\\n", $0}'  > concatenated.pem
   

3. To give the CockroachDB cluster access to your Azure tenant, create a Microsoft Entra application. When registering the application, leave the optional Redirect URI empty since certificate-based authentication is being used, not browser-based authentication.

   1. Once the application is registered, upload the certificate file (ca.crt) created in step 2.
   2. From the Overview page for the Entra application, note the values for Display Name, Application (client) ID, and Directory (tenant) ID, which will be used in step 5.

4. To allow Azure Monitor to receive metrics, create an Application Insights resource.

   1. In the newly created Application Insights resource, add a role assignment that assigns the Entra application (search for the Display Name from step 3) to the role Monitoring Metrics Publisher.
   2. On the Overview page for the Application Insights resource, view the Connection String. Note the values for InstrumentationKey and IngestionEndpoint, which will be used in step 5.

5. To enable metrics export for your CockroachDB Advanced cluster, issue the following Cloud API command:

   curl --request POST \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/azuremonitor \
     --header "Authorization: Bearer {secret_key}" \
     --data '{ "application_id": "{application_id}", "tenant_id": "{tenant_id}", "ingestion_endpoint": "{ingestion_endpoint}", "instrumentation_key": "{instrumentation_key}", "certificate": "{combined_certificate_and_key}" }'
   

   Where:

   • {cluster_id} is your CockroachDB Advanced cluster ID from step 1.
   • {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.
   • {application_id} is the ID of the Entra application, Application (client) ID from step 3.
   • {tenant_id} is the ID of the tenant where the Entra application was created, Directory (tenant) ID from step 3.
   • {ingestion_endpoint} is the target host where the metrics are to be sent, IngestionEndpoint from step 4.
   • {instrumentation_key} is the ID of the Azure Application Insights resource, InstrumentationKey from step 4.
   • {combined_certificate_and_key} is the string in the concatenated file from step 2.

6. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/azuremonitor \
     --header "Authorization: Bearer {secret_key}"
   

   When the command returns a status of ENABLED, the configuration has been applied to all nodes, and metrics will begin appearing in Azure Monitor. Since the configuration is applied to cluster nodes one at a time, metrics may begin to appear even before the status is ENABLED.

7. Once metrics export has been enabled, you can access metrics from your CockroachDB Advanced cluster directly in Azure Portal:

   1. Navigate to the Application Insights resource created in step 4.
   2. In the left menu under Monitoring, click Metrics. In the chart options, in the Metric search field, enter crdb_dedicated. This should retrieve a dropdown list of the exported metrics. Select a metric such as crdb_dedicated.capacity from the dropdown list.
   3. To further verify the metrics export, click Apply splitting, select cluster from the Values dropdown list. Your cluster name should appear in the legend under the chart.

To check the status of an existing Amazon CloudWatch metrics export configuration, use the following Cloud API command:

curl --request GET \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/cloudwatch \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To check the status of an existing Datadog metrics export configuration, use the following Cloud API command:

curl --request GET \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/datadog \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To check the status of an existing Prometheus metrics export configuration, use the following Cloud API command:

curl --request GET \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/prometheus \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To check the status of an existing Azure Monitor metrics export configuration, use the following Cloud API command:

curl --request GET \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/azuremonitor \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To disable an existing Amazon CloudWatch metrics export configuration, and stop sending metrics to CloudWatch, use the following Cloud API command:

curl --request DELETE \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/cloudwatch \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To disable an existing Datadog metrics export configuration, and stop sending metrics to Datadog, use the following Cloud API command:

curl --request DELETE \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/datadog \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To disable an existing Prometheus metrics export configuration, and stop sending metrics to Prometheus, use the following Cloud API command:

curl --request DELETE \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/prometheus \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.

To disable an existing Azure Monitor metrics export configuration, and stop sending metrics to Azure Monitor, use the following Cloud API command:

curl --request DELETE \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport/azuremonitor \
  --header "Authorization: Bearer {secret_key}"

Where:

• {cluster_id} is your CockroachDB Advanced cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
• {secret_key} is your CockroachDB Advanced API key. See API Access for instructions on generating this key.