Debug & Error Logs

On this page Carat arrow pointing down
Warning:
As of November 10, 2018, CockroachDB v1.0 is no longer supported. For more details, refer to the Release Support Policy.

If you need to troubleshoot issues with your cluster, you can check a node's logs, which include details about certain node-level and range-level events, such as errors. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem.

Details

When a node processes a cockroach command, it produces a stream of messages about the command's activities. Each message's body describes the activity, and its envelope contains metadata such as the message's severity level.

As a command generates messages, CockroachDB uses the command's logging flags and the message's severity level to determine the appropriate location for it.

Each node's logs detail only the internal activity of that node without visibility into the behavior of other nodes in the cluster. When troubleshooting, this means that you must identify the node where the problem occurred or collect the logs from all active nodes in your cluster.

Note:
You can also log queries your cluster receives.

Commands

All cockroach commands support logging. However, it's important to note:

  • cockroach start generates most messages related to the operation of your cluster.
  • Other commands do generate messages, but they're typically only interesting in troubleshooting scenarios.

Severity Levels

CockroachDB identifies each message with a severity level, letting operators know if they need to intercede:

  1. INFO (lowest severity; no action necessary)
  2. WARNING
  3. ERROR
  4. FATAL (highest severity; requires operator attention)

Default Behavior by Severity Level

Command INFO messages WARNING and above messages
cockroach start Write to file Write to file
All other commands Discard Print to stderr

Output Locations

Based on the command's flags and the message's severity level, CockroachDB does one of the following:

Write to File

CockroachDB can write messages to log files, which use the following format:

cockroach.[host].[user].[start timestamp in system time].[process ID].log
Property cockroach start All other commands
Enabled by Default1 Explicit --log-dir flag
Default File Destination [firststoredir]/logs N/A
Change File Destination --log-dir=[destination] --log-dir=[destination]
Default Severity Level Threshold INFO N/A
Change Severity Threshold --log-file-verbosity=[severity level] --log-file-verbosity=[severity level]
Disabled by --log-dir=1 Default
Note:
1 If the cockroach process does not have access to on-disk storage, cockroach start does not write messages to log files; instead it prints all messages to stderr.

CockroachDB can print messages to stderr, which normally prints them to the machine's terminal but does not store them.

Property cockroach start All other commands
Enabled by Explicit --logtostderr flag2 Default
Default Severity Level Threshold N/A WARNING
Change Severity Threshold --logtostderr=[severity level] --logtostderr=[severity level]
Disabled by Default2 --logtostderr=NONE
Note:
2 cockroach start does not print any messages to stderr unless the cockroach process does not have access to on-disk storage, in which case it defaults to --logtostderr=INFO and prints all messages to stderr.

Discard Message

Messages with severity levels below the --logtostderr and --log-file-verbosity flag's values are neither written to files nor printed to stderr, so they are discarded.

By default, commands besides cockroach start discard messages with the INFO severity level.

Flags

Flag Description
--log-dir Enable logging to files and write logs to the specified directory.

Setting --log-dir to a blank directory (--log-dir=) disables logging to files. Do not use --log-dir=""; this creates a new directory named "" and stores log files in that directory.
--log-dir-max-size After the log directory reaches the specified size, delete the oldest log file. The flag's argument takes standard file sizes, such as --log-dir-max-size=1GiB.

Default: 100MiB
--log-file-max-size After logs reach the specified size, begin writing logs to a new file. The flag's argument takes standard file sizes, such as --log-file-max-size=2MiB.

Default: 10MiB
--log-file-verbosity Only writes messages to log files if they are at or above the specified severity level, such as --log-file-verbosity=WARNING. Requires logging to files.

Default: INFO
--logtostderr Enable logging to stderr for messages at or above the specified severity level, such as --logtostderr=ERROR

If you use this flag without specifying the severity level (e.g., cockroach start --logtostderr), it prints messages of all severities to stderr.

Setting --logtostderr=NONE disables logging to stderr.
--no-color Do not colorize stderr. Possible values: true or false.

When set to false, messages logged to stderr are colorized based on severity level.

Default: false

The --log-backtrace-at, --verbosity, and --v flags are intended for internal debugging by CockroachDB contributors.

Log Queries

To help troubleshoot query performance issues, you can use cluster-wide settings to enable logging for long-running SQL transactions or all queries, regardless of time.

Warning:
These settings makes all queries slower and causes nodes to consume more memory. You should disable query logging as soon as you're done troubleshooting the query's issues.

Enable Query Logging

  • Long-running transactions:

    > SET CLUSTER SETTING sql.trace.txn.enable_threshold = '[time]';
    

    The [time] parameter accepts common time specifiers, such as 100ms or 2s.

  • All queries:

    > SET CLUSTER SETTING sql.trace.log_statement_execute = true;
    

Details

After you enable query logging, whenever nodes process SQL statements, they generate messages with an INFO severity level.

By default, these messages will get written to files, but are ultimate handled by whatever logging behavior you set for cockroach start.

Improve Query Performance

After finding which queries are slow, use EXPLAIN to examine them. It's possible that the query is slow because it performs a full-table scan. In these cases, you can likely improve the query's performance by adding an index.

(More guidance around query performance optimization forthcoming.)

Disable Query Logging

Once you're done troubleshooting, you should disable query logging to prevent it from unnecessarily consuming resources.

  • Long-running transactions:

    > SET CLUSTER SETTING sql.trace.txn.enable_threshold = '0s';
    
  • All queries:

    > SET CLUSTER SETTING sql.trace.log_statement_execute = false;
    

See Also


Yes No
On this page

Yes No