Physical Cluster Replication Monitoring

On this page

New in v23.2: You can monitor a physical cluster replication stream using:

• SHOW VIRTUAL CLUSTER ... WITH REPLICATION STATUS in the SQL shell.
• The Physical Replication dashboard on the DB Console.
• Prometheus and Alertmanager to track and alert on replication metrics.

When you complete a cutover, there will be a gap in the primary cluster's metrics whether you are monitoring via the DB Console or Prometheus.

The standby cluster will also require separate monitoring to ensure observability during the cutover period. You can use the DB console to track the relevant metrics, or you can use a tool like Grafana to create two separate dashboards, one for each cluster, or a single dashboard with data from both clusters.

SQL Shell

In the standby cluster's SQL shell, you can query SHOW VIRTUAL CLUSTER ... WITH REPLICATION STATUS for detail on status and timestamps for planning cutover:

SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;

Refer to Responses for a description of each field.

id |        name        |     data_state     | service_mode | source_tenant_name |                                                     source_cluster_uri                                               | replication_job_id |        replicated_time        |         retained_time         | cutover_time
---+--------------------+--------------------+--------------+--------------------+----------------------------------------------------------------------------------------------------------------------+--------------------+-------------------------------+-------------------------------+---------------
3  | application        | replicating        | none         | application        | postgresql://{user}:{password}@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 899090689449132033 | 2023-09-11 22:29:35.085548+00 | 2023-09-11 16:51:43.612846+00 |     NULL
(1 row)

Responses

Data state

DB Console

You can access the DB Console for your standby cluster at https://{your IP or hostname}:8080/. Select the Metrics page from the left-hand navigation bar, and then select Physical Cluster Replication from the Dashboard dropdown. The user that accesses the DB Console must have admin privileges to view this dashboard.

Use the Graph menu to display metrics for your entire cluster or for a specific node.

To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the SQL Activity pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.

Hovering your mouse pointer over the graph title will display a tooltip with a description and the metrics used to create the graph.

When hovering on graphs, crosshair lines will appear at your mouse pointer. The series' values corresponding to the given time in the cross hairs are displayed in the legend under the graph. Hovering the mouse pointer on a given series displays the corresponding value near the mouse pointer and highlights the series line (graying out other series lines). Click anywhere within the graph to freeze the values in place. Click anywhere within the graph again to cause the values to change with your mouse movements once more.

In the legend, click on an individual series to isolate it on the graph. The other series will be hidden, while the hover will still work. Click the individual series again to make the other series visible. If there are many series, a scrollbar may appear on the right of the legend. This is to limit the size of the legend so that it does not get endlessly large, particularly on clusters with many nodes.

The Physical Cluster Replication dashboard contains graphs for monitoring:

Logical bytes

The Logical Bytes graph shows you the throughput of the replicated bytes.

Hovering over the graph displays:

• The date and time.
• The number of logical bytes replicated in MiB.

SST bytes

The SST Bytes graph shows you the rate at which all SST bytes are sent to the KV layer by physical cluster replication jobs.

Hovering over the graph displays:

• The date and time.
• The number of SST bytes replicated in MiB.

Prometheus

You can use Prometheus and Alertmanager to track and alert on physical cluster replication metrics. Refer to the Monitor CockroachDB with Prometheus tutorial for steps to set up Prometheus.

We recommend tracking the following metrics:

• physical_replication.logical_bytes: The logical bytes (the sum of all keys and values) ingested by all physical cluster replication jobs.
• physical_replication.sst_bytes: The SST bytes (compressed) sent to the KV layer by all physical cluster replication jobs.
• physical_replication.replicated_time_seconds: The replicated time of the physical replication stream in seconds since the Unix epoch.

Data verification

To verify that the data at a certain point in time is correct on the standby cluster, you can use the current replicated time from the replication job information to run a point-in-time fingerprint on both the primary and standby clusters. This will verify that the transmission and ingestion of the data on the standby cluster, at that point in time, is correct.

1. Retrieve the current replicated time of the replication job on the standby cluster with SHOW VIRTUAL CLUSTER:

   SELECT replicated_time FROM [SHOW VIRTUAL CLUSTER standbyapplication WITH REPLICATION STATUS];
   

        replicated_time
   ----------------------------
   2024-01-09 16:15:45.291575+00
   (1 row)
   

   For detail on connecting to the standby cluster, refer to Set Up Physical Cluster Replication.

2. From the primary cluster's system virtual cluster, specify a timestamp at or earlier than the current replicated_time to retrieve the fingerprint. This example uses the current replicated_time:

   SELECT * FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM VIRTUAL CLUSTER application] AS OF SYSTEM TIME '2024-01-09 16:15:45.291575+00';
   

   tenant_name |             end_ts             |     fingerprint
   ------------+--------------------------------+----------------------
   application | 1704816945291575000.0000000000 | 2646132238164576487
   (1 row)
   

   For detail on connecting to the primary cluster, refer to Set Up Physical Cluster Replication.

3. From the standby cluster's system virtual cluster, specify the same timestamp used on the primary cluster to retrieve the standby cluster's fingerprint:

   SELECT * FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM VIRTUAL CLUSTER standbyapplication] AS OF SYSTEM TIME '2024-01-09 16:15:45.291575+00';
   

       tenant_name     |             end_ts             |     fingerprint
   --------------------+--------------------------------+----------------------
   standbyapplication  | 1704816945291575000.0000000000 | 2646132238164576487
   (1 row)
   

4. Compare the fingerprints of the primary and standby clusters to verify the data. The same value for the fingerprints indicates the data is correct.

See also

• DB Console Overview
• Monitoring and Alerting
• Physical Cluster Replication Overview
• Cluster Virtualization Overview

Was this helpful?