cockroach debug zip

On this page

The cockroach debug zip command connects to your cluster and gathers information from each active node into a single .zip file (inactive nodes are not included). For details on the .zip contents, see Files.

You can use the cockroach debug merge-logs command in conjunction with cockroach debug zip to merge the collected logs into one file, making them easier to parse.

Details

Use cases

There are two scenarios in which debug zip is useful:

• If you experience severe or difficult-to-reproduce issues with your cluster, Cockroach Labs might ask you to send us your cluster's debugging information using cockroach debug zip. We recommend reducing the *.zip file size by only retrieving debugging information for the relevant time range of the issue by using the --files-from, and/or --files-until flags.

• To collect all of your nodes' logs, which you can then parse to locate issues. You can optionally use the flags to retrieve only the log files. For more information about logs, see Logging. Also note:

  • Nodes that are currently down cannot deliver their logs over the network. For these nodes, you must log on to the machine where the cockroach process would otherwise be running, and gather the files manually.
  • Nodes that are currently up but disconnected from other nodes (e.g., because of a network partition) may not be able to respond to debug zip requests forwarded by other nodes, but can still respond to requests for data when asked directly. In such situations, we recommend using the --host flag to point debug zip at each of the disconnected nodes until data has been gathered for the entire cluster.

Files

cockroach debug zip collects log files, heap profiles, CPU profiles, and goroutine dumps from the last 48 hours, by default.

The following files collected by cockroach debug zip, which are found in the individual node directories, can be filtered using the --exclude-files, --include-files, --files-from, and/or --files-until flags:

The following information is also contained in the .zip file, and cannot be filtered:

• System tables. The following system tables are not included:
  • system.users
  • system.web_sessions
  • system.join_tokens
  • system.comments
  • system.ui
  • system.zones
  • system.statement_bundle_chunks
  • system.statement_statistics
  • system.transaction_statistics
• Cluster events
• Database details
• Schema change events
• Database, table, node, and range lists
• Node details
• Node liveness
• Gossip data
• Stack traces
• Range details
• Jobs
• Cluster Settings
• Metrics
• Replication Reports
• CPU profiles
• A script (hot-ranges.sh) that summarizes the hottest ranges (ranges receiving a high number of reads or writes)

Subcommands

While the cockroach debug command has a few subcommands, users are expected to use only the zip, encryption-active-key, merge-logs, list-files, tsdump, and ballast subcommands.

We recommend using the encryption-decrypt and job-trace subcommands only when directed by the Cockroach Labs support team.

The other debug subcommands are useful only to Cockroach Labs. Output of debug commands may contain sensitive or secret information.

Synopsis

$ cockroach debug zip {ZIP file destination} {flags}

Flags

The debug zip subcommand supports the following general-use, client connection, and logging flags.

General

Client connection

Logging

By default, this command logs messages to stderr. This includes events with WARNING severity and higher.

If you need to troubleshoot this command's behavior, you can customize its logging behavior.

Examples

Generate a debug zip file

Generate the debug zip file for an insecure cluster:

$ cockroach debug zip ./cockroach-data/logs/debug.zip --insecure --host=200.100.50.25

Generate the debug zip file for a secure cluster:

$ cockroach debug zip ./cockroach-data/logs/debug.zip --host=200.100.50.25

Generate a debug zip file for a time range

Generate a debug zip file containing only debugging information for a specified time range:

$ cockroach debug zip ./cockroach-data/logs/debug.zip --files-from='2023-10-03 13:30' --files-until='2023-10-03 14:30'

Generate a debug zip file with logs only

Generate a debug zip file containing only log files:

$ cockroach debug zip ./cockroach-data/logs/debug.zip --include-files=*.log

Redact sensitive information

Log redaction

Example of a log string without redaction enabled:

server/server.go:1423 ⋮ password of user ‹admin› was set to ‹"s3cr34?!@x_"›

Enable log redaction:

$ cockroach debug zip ./cockroach-data/logs/debug.zip --redact --insecure --host=200.100.50.25

server/server.go:1423 ⋮ password of user ‹×› was set to ‹×›

Hostname and IP address redaction

Example of status.json without hostname and IP address redaction enabled:

{
  "node_id": 1,
  "address": {
    "network_field": "tcp",
    "address_field": "200.100.50.25:26257"
  },
  "sql_address": {
    "network_field": "tcp",
    "address_field": "200.100.50.25:26257"
  }
}

First, enable the cluster setting:

SET CLUSTER SETTING debug.zip.redact_addresses.enabled = true;

Then, generate .zip with log redaction as well as hostname and IP address redaction:

cockroach debug zip ./cockroach-data/logs/debug.zip --redact --insecure --host=200.100.50.25

status.json with hostname and IP address redaction:

{
  "node_id": 1,
  "address": {
    "network_field": "tcp",
    "address_field": "‹×›"
  },
  "sql_address": {
    "network_field": "tcp",
    "address_field": "‹×›"
  }
}

See also

• File an Issue
• cockroach Commands Overview
• Troubleshooting Overview

Was this helpful?