Configure Logs

On this page

This page describes how to configure CockroachDB logs with the --log or log-config-file flag and a YAML payload. Most logging behaviors are configurable, including:

• The log sinks that output logs to different locations, including over the network.
• The logging channels that are mapped to each sink.
• The format used by the log messages.
• The redaction of log messages.
• The buffering of log messages.

For examples of how these settings can be used in practice, see Logging Use Cases.

Flag

To configure the logging behavior of a cockroach command, include one of these flags with the command:

• --log={yaml}, where {yaml} is the YAML payload
• --log-config-file={yaml-file}, where {yaml-file} is the path to a YAML file

To disable logging, set --log-dir to a blank directory (--log-dir=) instead of using one of the other logging flags. Do not use --log-dir=""; this creates a new directory named "" and stores log files in that directory.

YAML payload

All log settings for a cockroach command are specified with a YAML payload in one of the following formats:

• Block format, where each parameter is written on a separate line. For example, after creating a file logs.yaml, pass the YAML values with either --log-config-file or --log:

  $ cockroach start-single-node --certs-dir=certs --log-config-file=logs.yaml
  

  $ cockroach start-single-node --certs-dir=certs --log="$(cat logs.yaml)"
  

• Inline format, where all parameters are specified on one line. For example, to generate an ops log file that collects the OPS and HEALTH channels (overriding the file groups defined for those channels in the default configuration):

  $ cockroach start-single-node --certs-dir=certs --log="sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}"
  

  Note that the inline spaces must be preserved.

For clarity, this article uses the block format to describe the YAML payload, which has the overall structure:

file-defaults: ...       # defaults inherited by file sinks
fluent-defaults: ...     # defaults inherited by Fluentd sinks
http-defaults: ...       # defaults inherited by HTTP sinks
sinks:
   file-groups: ...      # file sink definitions
   fluent-servers: ...   # Fluentd sink definitions
   http-servers: ...     # HTTP sink definitions
   stderr: ...           # stderr sink definitions
capture-stray-errors: ... # parameters for the stray error capture system

Providing a logging configuration is optional. Any fields included in the YAML payload will override the same fields in the default logging configuration.

Configure log sinks

Log sinks route events from specified logging channels to destinations outside CockroachDB. These destinations currently include log files, Fluentd-compatible servers, HTTP servers, and the standard error stream (stderr).

All supported output destinations are configured under sinks:

file-defaults: ...
fluent-defaults: ...
http-defaults: ...
sinks:
  file-groups:
    {file group name}:
      channels: {channels}
      ...
  fluent-servers:
    {server name}:
      channels: {channels}
      ...
  http-servers:
    {server name}:
      channels: {channels}
      ...
  stderr:
      channels: {channels}
      ...

All supported sink types use the following common sink parameters:

If not specified for a given sink, these parameter values are inherited from file-defaults (for file sinks), fluent-defaults (for Fluentd sinks), and http-defaults (for HTTP sinks).

Output to files

CockroachDB can write messages to one or more log files.

file-groups specifies the channels that output to each log file, along with its output directory and other configuration details. For example:

file-defaults: ...
fluent-defaults: ...
http-defaults: ...
sinks:
  file-groups:
    default:
      channels: [DEV]
    health:
      channels: [HEALTH]
      dir: health-logs
    ...

Along with the common sink parameters, each file group accepts the following additional parameters:

If not specified for a given file group, the parameter values are inherited from file-defaults (except channels, which uses the default configuration).

Log file naming

Log files are named in the following format:

{process}-{file group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log

For example, a file group health will generate a log file that looks like this:

cockroach-health.work-laptop.worker.2021-03-15T15_24_10Z.024338.log

For each file group, a symlink points to the latest generated log file. It's easiest to refer to the symlink. For example:

cockroach-health.log

Access in DB Console

Known limitations

Log files can only be accessed in the DB Console if they are stored in the same directory as the file sink for the DEV channel.

Output to Fluentd-compatible network collectors

CockroachDB can send logs over the network to a Fluentd-compatible log collector (e.g., Elasticsearch, Splunk). fluent-servers specifies the channels that output to a server, along with the server configuration details. For example:

file-defaults: ...
fluent-defaults: ...
http-defaults: ...
sinks:
  fluent-servers:
    health:
      channels: [HEALTH]
      address: 127.0.0.1:5170
    ...

Along with the common sink parameters, each Fluentd server accepts the following additional parameters:

For an example network logging configuration, see Logging use cases.

Output to HTTP network collectors

CockroachDB can send logs over the network to an HTTP server. http-servers specifies the channels that output to a server, along with the server configuration details. For example:

file-defaults: ...
fluent-defaults: ...
http-defaults: ...
sinks:
  http-servers:
    health:
      channels: [HEALTH]
      address: 127.0.0.1:5170
      method: POST
      unsafe-tls: false
      timeout: 2s
      disable-keep-alives: false
      ...

Along with the common sink parameters, each HTTP server accepts the following additional parameters:

For an example network logging configuration, see Logging use cases. For an example that uses compression and headers parameters, see Configure an HTTP network collector for Datadog.

Output to stderr

CockroachDB can output messages to the standard error stream (stderr), which prints them to the machine's terminal but does not store them. stderr specifies the channels that output to the stream. For example:

file-defaults: ...
fluent-defaults: ...
http-defaults: ...
sinks:
  stderr:
    channels: [DEV]

Along with the common sink parameters, stderr accepts the following additional parameters:

Because server start-up messages are always emitted at the start of the standard error stream, it is generally difficult to automate integration of stderr with log analyzers. We recommend using file logging or network logging via Fluentd or HTTP instead of stderr when integrating with automated monitoring software.

Logging channel selection

For each sink, multiple channels can be selected. Note that:

• Spacing between items will vary according to the syntax used.
• Channel selection is case-insensitive.

These selections are equivalent:

# Select OPS and HEALTH.
channels: [OPS, HEALTH]
channels: 'OPS, HEALTH'
channels: OPS,HEALTH
channels:
- OPS
- HEALTH

By default, the severity level set by filter in the sink configuration is used. However, you can specify a different severity level for each channel. For more information on severity levels, see Set logging levels.

These selections are equivalent:

# Select PERF at severity INFO, and HEALTH and OPS at severity WARNING.
channels: {INFO: [PERF], WARNING: [HEALTH, OPS]}
channels:
  INFO:
  - PERF
  WARNING:
  - OPS
  - HEALTH

Brackets are optional when selecting a single channel:

channels: OPS
channels: {INFO: PERF}

To select all channels, using the all keyword:

channels: all
channels: 'all'
channels: [all]
channels: ['all']

To select all channels except for a subset, using the all except keyword prefix:

channels: all except ops,health
channels: all except [ops,health]
channels: 'all except ops, health'
channels: 'all except [ops, health]'

Configure logging defaults

When setting up a logging configuration, it's simplest to define shared parameters in file-defaults and fluent-defaults and override specific values as needed in file-groups, fluent-servers, http-servers, and stderr. For a complete example, see the default configuration.

This section describes some recommended defaults.

Set file defaults

Defaults for log files are set in file-defaults, which accepts all common sink parameters and the file group parameters dir, max-file-size, max-group-size, file-permissions, and buffered-writes.

Logging directory

By default, CockroachDB adds log files to a logs subdirectory in the first on-disk store directory (default: cockroach-data):

cockroach-data/logs

In cloud deployments, the main data store will be subject to an IOPS budget. Adding logs to the store directory will excessively consume IOPS. For this reason, cloud deployments should output log files to a separate directory with fewer IOPS restrictions.

You can override the default logging directory like this:

file-defaults:
  dir: /custom/dir/path/

File logging format

The default message format for log files is crdb-v2. Each crdb-v2 log message starts with a flat prefix that contains event metadata (e.g., severity, date, timestamp, channel), followed by the event payload. For details on the metadata, see Log formats.

If you plan to read logs programmatically, you can switch to a JSON or compact JSON format:

file-defaults:
  format: json

Set Fluentd defaults

Defaults for Fluentd-compatible network sinks are set in fluent-defaults, which accepts all common sink parameters.

Note that the server parameters address and net are not specified in fluent-defaults:

• address must be specified for each sink under fluent-servers.
• net is not required and defaults to tcp.

Fluentd logging format

The default message format for network output is json-fluent-compact. Each log message is structured as a JSON payload that can be read programmatically. The json-fluent-compact and json-fluent formats include a tag field that is required by the Fluentd protocol. For details, see Log formats.

fluent-defaults:
  format: json-fluent

Set HTTP defaults

Defaults for HTTP sinks are set in http-defaults, which accepts all common sink parameters.

Note that the server parameters address and method are not specified in fluent-defaults:

• address must be specified for each sink under http-servers.
• method is not required and defaults to POST.

HTTP logging format

The default message format for HTTP output is json-compact. Each log message is structured as a JSON payload that can be read programmatically. For details, see Log formats.

http-defaults:
  format: json

Set logging levels

Log messages are associated with a severity level when they are generated.

Each logging sink accepts messages from each logging channel at a minimum severity level. This minimum severity level can be specified per sink or by default using the filter attribute.

Messages with severity levels below the configured threshold do not enter logging channels and are discarded.

The default configuration uses the following severity levels for cockroach start and cockroach start-single-node:

• file-defaults, fluent-defaults, and http-defaults each use filter: INFO. Since INFO is the lowest severity level, file and network sinks will emit all log messages.
• stderr uses filter: NONE and does not emit log messages.
• The default file group uses filter: INFO for events from the DEV and OPS channels, and filter: WARNING for all others.

You can override the file-defaults, fluent-defaults, and http-defaults severity levels on a per-sink basis.

For example, this will include DEV events at severity WARNING:

sinks:
  file-groups:
    dev:
      channels: DEV
      filter: WARNING

You can also override the filter attribute and set severity levels on a per-channel basis.

For example, this will include DEV events at severity INFO, and OPS events at severity ERROR:

sinks:
  file-groups:
    dev:
      channels: {INFO: DEV, ERROR: OPS}

Redact logs

CockroachDB can redact personally identifiable information (PII) from log messages. The logging system includes two parameters that handle this differently:

• redact is disabled by default. When enabled, redact automatically redacts sensitive data from logging output. We do not recommend enabling this on the DEV channel because it impairs our ability to troubleshoot problems.
• redactable is enabled by default. This places redaction markers around sensitive fields in log messages. These markers are recognized by cockroach debug zip and cockroach debug merge-logs, which aggregate CockroachDB log files and can be instructed to redact sensitive data from their output.

When collecting logs centrally (e.g., in data mining scenarios where non-privileged users have access to logs) or over a network (e.g., to an external log collector), it's safest to enable redact:

file-defaults:
  redact: true
fluent-defaults:
  redact: true
http-defaults:
  redact: true

External log collectors can misinterpret the cockroach debug redaction markers (< >), since they are specific to CockroachDB. To prevent this issue when using network sinks, disable redactable:

fluent-defaults:
  redactable: false

If the default redaction behavior and policies do not meet redaction requirements, we recommend using the external log collectors with the redaction markers (< >) to redact. In this case, enable redactable.

DEV channel

The DEV channel is used for debug and uncategorized messages. It can therefore be noisy and contain sensitive (PII) information.

We recommend configuring DEV separately from the other logging channels. When sending logs to a Fluentd-compatible or HTTP network collector, DEV logs should also be excluded from network collection.

In this example, the dev file group is reserved for DEV logs. These are output to a cockroach-dev.log file in a custom disk dir:

file-defaults: ...
fluent-defaults: ...
sinks:
  file-groups:
    dev:
      channels: [DEV]
      dir: /custom/dir/path/
    ...

Log buffering for network sinks

Both Fluentd-compatible and HTTP log sinks support the buffering of log messages by default. Previous to version v22.2, log buffering was only available for the log file log sink.

With log buffering configured, log messages are held in a buffer for a configurable time period or accumulated message size threshold before being written to the target log sink together as a batch. Log buffering helps to ensure consistent low-latency log message writes over the network even in high-traffic, high-contention scenarios.

The following shows a basic log configuration with buffering configured for a Fluentd-compatible log sink:

fluent-defaults:
  buffering:
    max-staleness: 20s
    flush-trigger-size: 2MiB
    max-buffer-size: 100MiB
sinks:
  fluent-servers:
    health:
      channels: HEALTH
      buffering:
        max-staleness: 2s  # Override max-staleness for HEALTH channel only

With this logging configuration:

• CockroachDB will hold log messages in a buffer for up to 20s (the max-staleness setting), or up to a collected size of 2MiB (the flush-trigger-size setting), before writing ("flushing") the buffer to the log file. When both settings are used, whichever case is met first triggers the buffer flush.
• If at any point the accumulated message size of buffer flushes (as triggered by reaching either the configured max-staleness or flush-trigger-size value) exceeds 100MiB (the max-buffer-size setting), all new incoming log messages received are dropped until the accumulated message size in the buffer once more falls below this value.
• For the HEALTH log channel only, override the file-defaults value of 20s for max-staleness, instead flushing messages to the log file within up to 2s.

Alternatively, you may explicitly disable log buffering by setting buffering to NONE. The following log configuration explicitly disables log buffering for just the OPS channel on a Fluentd-compatible log sink:

file-defaults: ...
fluent-defaults: ...
sinks:
  fluent-servers:
    ops:
      channels: OPS
      buffering: NONE

The following disables log buffering completely for the Fluentd-compatible sink:

fluent-defaults:
  buffering: NONE

Stray error capture

Certain events, such as uncaught software exceptions (panics), bypass the CockroachDB logging system. However, they can be useful in troubleshooting. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem.

To ensure that these stray errors can be tracked, CockroachDB does not send them to stderr by default. Instead, stray errors are output to a cockroach-stderr.log file in the default logging directory.

You can change these settings in capture-stray-errors:

file-defaults: ...
fluent-defaults: ...
sinks: ...
capture-stray-errors:
  enable: true
  dir: /custom/dir/path/

Default logging configuration

The YAML payload below represents the default logging behavior of cockroach start and cockroach start-single-node.

file-defaults:
  max-file-size: 10MiB
  max-group-size: 100MiB
  file-permissions: 644
  buffered-writes: true
  filter: INFO
  format: crdb-v2
  redact: false
  redactable: true
  exit-on-error: true
  auditable: false
fluent-defaults:
  filter: INFO
  format: json-fluent-compact
  redact: false
  redactable: true
  exit-on-error: false
  auditable: false
http-defaults:
  method: POST
  unsafe-tls: false
  timeout: 0s
  disable-keep-alives: false
  filter: INFO
  format: json-compact
  redact: false
  redactable: true
  exit-on-error: false
  auditable: false
sinks:
  file-groups:
    default:
      channels:
        INFO: [DEV, OPS]
        WARNING: all except [DEV, OPS]
    health:
      channels: [HEALTH]
    kv-distribution:
      channels: [KV_DISTRIBUTION]
    pebble:
      channels: [STORAGE]
    security:
      channels: [PRIVILEGES, USER_ADMIN]
      auditable: true
    sql-audit:
      channels: [SENSITIVE_ACCESS]
      auditable: true
    sql-auth:
      channels: [SESSIONS]
      auditable: true
    sql-exec:
      channels: [SQL_EXEC]
    sql-slow:
      channels: [SQL_PERF]
    sql-slow-internal-only:
      channels: [SQL_INTERNAL_PERF]
    telemetry:
      channels: [TELEMETRY]
      max-file-size: 100KiB
      max-group-size: 1.0MiB
  stderr:
    channels: all
    filter: NONE
    redact: false
    redactable: true
    exit-on-error: true
capture-stray-errors:
  enable: true
  max-group-size: 100MiB

Note that a default dir is not specified for file-defaults and capture-stray-errors:

• The default dir for file-defaults is inferred from the first on-disk store directory. See Logging directory.
• The default dir for capture-stray-errors is inherited form file-defaults.

See also

• Logging Use Cases
• Log Formats

Was this helpful?