The Queues dashboard lets you monitor the health and performance of various queueing systems in CockroachDB.
To view this dashboard, access the DB Console and click Metrics on the left-hand navigation, and then select Dashboard > Queues.
Dashboard navigation
Use the Graph menu to display metrics for your entire cluster or for a specific node.
To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the SQL Activity pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.
Hovering your mouse pointer over the graph title will display a tooltip with a description and the metrics used to create the graph.
When hovering on graphs, crosshair lines will appear at your mouse pointer. The series' values corresponding to the given time in the cross hairs are displayed in the legend under the graph. Hovering the mouse pointer on a given series displays the corresponding value near the mouse pointer and highlights the series line (graying out other series lines). Click anywhere within the graph to freeze the values in place. Click anywhere within the graph again to cause the values to change with your mouse movements once more.
In the legend, click on an individual series to isolate it on the graph. The other series will be hidden, while the hover will still work. Click the individual series again to make the other series visible. If there are many series, a scrollbar may appear on the right of the legend. This is to limit the size of the legend so that it does not get endlessly large, particularly on clusters with many nodes.
All timestamps in the DB Console are shown in Coordinated Universal Time (UTC).
The Queues dashboard displays the following time series graphs:
Queue Processing Failures
The Queue Processing Failures graph displays processing failures experienced across various queuing systems.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| GC | The number of replicas which failed processing in the garbage collection queue, as tracked by the queue.gc.process.failure metric. |
| Replica GC | The number of replicas which failed processing in the replica garbage collection queue, as tracked by the queue.replicagc.process.failure metric. |
| Replication | The number of replicas which failed processing in the replica queue, as tracked by the queue.replicate.process.failure metric. |
| Split | The number of replicas which failed processing in the split queue, as tracked by the queue.split.process.failure metric. |
| Consistency | The number of replicas which failed processing in the consistency checker queue, as tracked by the queue.consistency.process.failure metric. |
| Raft Log | The number of replicas which failed processing in the Raft log queue, as tracked by the queue.raftlog.process.failure metric. |
| Raft Snapshot | The number of replicas which failed processing in the Raft repair queue, as tracked by the queue.raftsnapshot.process.failure metric. |
| Time Series Maintenance | The number of replicas which failed processing in the time series maintenance queue, as tracked by the queue.tsmaintenance.process.failure metric. |
Queue Processing Times
The Queue Processing Times graph displays the processing rate across various queue systems.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| GC | The processing rate for the garbage collection queue, as tracked by the queue.gc.processingnanos metric. |
| Replica GC | The processing rate for the replica garbage collection queue, as tracked by the queue.replicagc.processingnanos metric. |
| Replication | The processing rate for the replication queue, as tracked by the queue.replicate.processingnanos metric. |
| Split | The processing rate for the split queue, as tracked by the queue.split.processingnanos metric. |
| Consistency | The processing rate for the consistency checker queue, as tracked by the queue.consistency.processingnanos metric. |
| Raft Log | The processing rate for the Raft log queue, as tracked by the queue.raftlog.processingnanos metric. |
| Raft Snapshot | The processing rate for the Raft repair queue, as tracked by the queue.raftsnapshot.processingnanos metric. |
| Time Series Maintenance | The processing rate for the time series maintenance queue, as tracked by the queue.tsmaintenance.processingnanos metric. |
Replica GC Queue
The Replica GC Queue graph displays various details about the health and performance of the replica garbage collection queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the replica garbage collection queue, as tracked by the queue.replicagc.process.success metric. |
| Pending Actions | The number of pending replicas in the replica garbage collection queue, as tracked by the queue.replicagc.pending metric. |
| Replicas Removed | The number of replica removals attempted in the replica garbage collection queue, as tracked by the queue.replicagc.removereplica metric. |
Replication Queue
The Replication Queue graph displays various details about the health and performance of the replication queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the replication queue, as tracked by the queue.replicate.process.success metric. |
| Pending Actions | The number of pending replicas in the replication queue, as tracked by the queue.replicate.pending metric. |
| Replicas Added | The number of replica additions attempted by the replication queue, as tracked by the queue.replicate.addreplica metric. |
| Replicas Removed | The number of replica removals attempted by the replication queue, as tracked by the queue.replicate.removereplica metric. |
| Dead Replicas Removed | The number of dead replica removals attempted by the replication queue, as tracked by the queue.replicate.removedeadreplica metric. |
| Learner Replicas Removed | The number of learner replica removals attempted by the replication queue, as tracked by the queue.replicate.removelearnerreplica metric. |
| Replicas Rebalanced | The number of replica rebalancer-initiated additions attempted by the replication queue, as tracked by the queue.replicate.rebalancereplica metric. |
| Leases Transferred | The number of range lease transfers attempted by the replication queue, as tracked by the queue.replicate.transferlease metric. |
| Replicas in Purgatory | The number of replicas in the replication queue's purgatory currently awaiting allocation options, as tracked by the queue.replicate.purgatory metric. |
Split Queue
The Split Queue graph displays various details about the health and performance of the split queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the split queue, as tracked by the queue.split.process.success metric. |
| Pending Actions | The number of pending replicas in the split queue, as tracked by the queue.split.pending metric. |
Merge Queue
The Merge Queue graph displays various details about the health and performance of the merge queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the merge queue, as tracked by the queue.merge.process.success metric. |
| Pending Actions | The number of pending replicas in the merge queue, as tracked by the queue.merge.pending metric. |
Raft Log Queue
The Raft Log Queue graph displays various details about the health and performance of the Raft log queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the Raft log queue, as tracked by the queue.raftlog.process.success metric. |
| Pending Actions | The number of pending replicas in the Raft log queue, as tracked by the queue.raftlog.pending metric. |
Raft Snapshot Queue
The Raft Snapshot Queue graph displays various details about the health and performance of the Raft repair queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the Raft repair queue, as tracked by the queue.raftsnapshot.process.success metric. |
| Pending Actions | The number of pending replicas in the Raft repair queue, as tracked by the queue.raftsnapshot.pending metric. |
Consistency Checker Queue
The Consistency Checker Queue graph displays various details about the health and performance of the consistency checker queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the consistency checker queue, as tracked by the queue.consistency.process.success metric. |
| Pending Actions | The number of pending replicas in the consistency checker queue, as tracked by the queue.consistency.pending metric. |
Time Series Maintenance Queue
The Time Series Maintenance Queue graph displays various details about the health and performance of the time series maintenance queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the time series maintenance queue, as tracked by the queue.tsmaintenance.process.success metric. |
| Pending Actions | The number of pending replicas in the time series maintenance queue, as tracked by the queue.tsmaintenance.pending metric. |
MVCC GC Queue
The MVCC GC Queue graph displays various details about the health and performance of the garbage collection queue.
Hovering over the graph displays values for the following metrics:
| Metric | Description |
|---|---|
| Successful Actions | The number of replicas successfully processed by the garbage collection queue, as tracked by the queue.gc.process.success metric. |
| Pending Actions | The number of pending replicas in the garbage collection queue, as tracked by the queue.gc.pending metric. |
Protected Timestamp Records
The Protected Timestamp Records graph displays the number of protected timestamp records (used by backups, changefeeds, etc. to prevent MVCC GC) per node, as tracked by the spanconfig.kvsubscriber.protected_record_count metric.
Summary and events
Summary panel
A Summary panel of key metrics is displayed to the right of the timeseries graphs.
| Metric | Description |
|---|---|
| Total Nodes | The total number of nodes in the cluster. Decommissioned nodes are not included in this count. |
| Capacity Used | The storage capacity used as a percentage of usable capacity allocated across all nodes. |
| Unavailable Ranges | The number of unavailable ranges in the cluster. A non-zero number indicates an unstable cluster. |
| Queries per second | The total number of SELECT, UPDATE, INSERT, and DELETE queries executed per second across the cluster. |
| P99 Latency | The 99th percentile of service latency. |
If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is not recommended in production), you must explicitly set the store size per node in order to display the correct capacity. Otherwise, the machine's actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.
Events panel
Underneath the Summary panel, the Events panel lists the 5 most recent events logged for all nodes across the cluster. To list all events, click View all events.
The following types of events are listed:
- Database created
- Database dropped
- Table created
- Table dropped
- Table altered
- Index created
- Index dropped
- View created
- View dropped
- Schema change reversed
- Schema change finished
- Node joined
- Node decommissioned
- Node restarted
- Cluster setting changed
See also
Was this helpful?
