What's New in v23.2

Docker image

Multi-platform images include support for both Intel and ARM. Multi-platform images do not take up additional space on your Docker host.

Within the multi-platform image:

• The ARM image is Experimental and not yet qualified for production use and not eligible for support or uptime SLA commitments.
• The Intel image is Generally Available for production use.

To download the Docker image:

docker pull cockroachdb/cockroach-unstable:v23.2.0-alpha.1

Source tag

To view or download the source code for CockroachDB v23.2.0-alpha.1 on Github, visit v23.2.0-alpha.1 source tag.

Backward-incompatible changes

• The pre-v23.1 output produced by SHOW RANGES, crdb_internal.ranges, and crdb_internal.ranges_no_leases was deprecated in 23.1 and is now replaced by default with output that's compatible with coalesced ranges (i.e., ranges that pack multiple tables/indexes/partitions into individual ranges). See the v23.1 release notes for SHOW RANGES for more details. #102961
• When a deployment is configured to use a time zone (new feature) for log file output using formats crdb-v1 or crdb-v2, it becomes impossible to process the new output log files using the cockroach debug merge-logs command from a previous version. The newest cockroach debug merge-logs code must be used instead. #104265
• When customizing the SQL shell's interactive prompt, the special sequence %M now expands to the full host name instead of the combination of host name and port number. To include the port number explicitly, use %>. The special sequence %m now expands to the host name up to the first period. #105137
• The cockroach debug zip command stores data retrieved from SQL tables in the remote cluster using the TSV format by default. #107474
• The changefeed.protect_timestamp.max_age cluster setting will only apply to newly created changefeeds in v23.2. For existing changefeeds, you can set the protect_data_from_gc_on_pause option so that changefeeds do not experience infinite retries and accumulate protected change data. You can use the ALTER CHANGEFEED statement to add protect_data_from_gc_on_pause to existing changefeeds. #103539
• The changefeed.new_pubsub_sink_enabled cluster setting is now enabled by default, which improves changefeed throughput. With this setting enabled, the top-level fields in JSON-encoded messages are capitalized: {Key: ..., Value: ..., Topic: ...}. After upgrading to CockroachDB v23.2, you may need to reconfigure downstream systems to parse the new message format. If you disable this setting, changefeeds emitting to Pub/Sub sinks with JSON-encoded events have the top-level message fields all lowercase: {key: ..., value: ..., topic: ...}.

Security updates

• Users who have the CREATEROLE role option can now grant and revoke role membership in any non-admin role. This change also removes the sql.auth.createrole_allows_grant_role_membership.enabled cluster setting, which was added in v23.1. In v23.2, the cluster setting is effectively always true. #104376

General changes

• You can now set Docker command arguments using the COCKROACH_ARGS environment variable. #98899
• Extended the /api/v2/nodes API endpoint with a storeMetrics field. #98208
• CockroachDB would previously use separate ranges for each table, index, or partition. This is no longer true. It is possible now to have multiple tables, indexes, and partitions get packed into the same range. For users with many of these schema objects, this will reduce the total range count in their clusters. This is especially true if individual tables, indexes, or partitions are smaller than the default configured maximum range size (controlled using zone configs, specifically the range_max_bytes parameter). We made this change to improve scalability with respect to the number of schema objects, since the underlying range count is now no longer a bottleneck. Users upgrading from v22.2, when finalizing their upgrade, may observe a round of range merges and snapshot transfers (to power said range merges) as a result of this change. If users want to opt-out of this optimization, they can configure the following cluster setting: SET CLUSTER SETTING spanconfig.storage_coalesce_adjacent.enabled = false; #98820
• EXPORT INTO PARQUET will now use a new internal implementation for writing Parquet files using the Parquet spec version 2.6. There should be no significant impact to the structure of files being written. There is one minor change: all columns written to Parquet files will be nullable (i.e., the Parquet repetition type is OPTIONAL). #104234
• Spatial libraries for CockroachDB now rely on GEOS 3.11 instead of GEOS 3.8. #106642
• CockroachDB no longer distributes libgeos for the experimental Windows build. Users can instead install GEOS directly from the source: https://libgeos.org/usage/download/. #106642
• The Formatting of byte figures in Pebble logs has been improved. Tools that parse these logs might need updating. #107392
• CockroachDB now has a new CLI option, --experimental-shared-storage to rebalance data faster from node to node. #105839
• Fixed a bug where, internally, if we print a 0 decimal with a very low exponent we use excessive memory. This is not possible when using the DECIMAL type, but may be possible when using crdb_internal functions. #110527

Enterprise edition changes

• The kafka_sink_config Compression and RequiredAcks options are now case-insensitive. #100929
• Changefeeds emit significantly fewer duplicate messages during node and cluster restarts. #102717
• CockroachDB has a new changefeed.protect_timestamp.max_age setting (by default 4 days), which will cancel running changefeed jobs if they fail to make forward progress for a period of time. This setting is used if the explicit gc_protect_expires_after option is not set. In addition, the protect_data_from_gc_on_pause option has been deprecated. This option is no longer needed since changefeed jobs always protect data. #103539
• Changefeeds now officially support the Parquet format using specification version 2.6. It is only usable with the cloud storage sink. The syntax to use Parquet is: CREATE CHANGEFEED FOR foo INTO ... WITH format=parquet. It supports all standard changefeed options and features including CDC transformations, except it does not support the topic_in_value option. #104528
• Changefeeds that create files over an HTTP connection may now be specified using INTO 'file-https://' to disambiguate with webhook-https. #107572
• The pgcrypto functions encrypt, encrypt_iv, decrypt, and decrypt_iv are now implemented. These functions require an enterprise license on a CCL distribution. #105654
• CockroachDB now paces the rangefeed goroutine creation rate to improve scheduler latency. This improves observability by adding an additional column in the crdb_internal.active_rangefeed table to indicate if the range is currently in catchup scan mode. #109346

SQL language changes

• Fixed the helper message on UPDATE SQL statements to include the optional FROM cause. #98709
• CockroachDB now supports enabling forward indexes and ordering on JSON values. #99275

• Added a new column visibility to crdb_internal.table_indexes and information_schema.statistics. Also added a new column visibility to the output of following SQL statements:

  SHOW INDEX FROM (table_name);
  

  SHOW INDEXES FROM (table_name);
  

  SHOW KEYS FROM (table_name);
  

  SHOW INDEX FROM DATABASE (database_name);
  

  SHOW INDEXES FROM DATABASE (database_name);
  

  SHOW KEYS FROM DATABASE (database_name);
  

  This new column contains a floating point number specifying the level of visibility of the index, from 0 (not visible) to 1 (fully visible). If the value is between 0 and 1, the index will be visible to the corresponding fraction of queries. #101334

• ALTER INDEX ... VISIBILITY ... is now supported. It can change an index visibility to any visibility between 0.0 and 1.0. Visibility 0.0 means the index is not visible to the optimizer, while visibility 1.0 means the index is fully visible. A value in the range between 0.0 and 1.0 means the index will be visible to the corresponding fraction of queries. #87301

• CockroachDB now has support for non-aggregate expressions involving columns outside of the grouping columns when the grouping columns include all key columns of a unique index and those key columns are not nullable. #101675

• CockroachDB now supports CREATE INDEX ... VISIBILITY ... and CREATE TABLE ... (... INDEX (...) VISIBILITY ...). This allows users to set the index visibility to any visibility between 0.0 and 1.0. Visibility 0.0 means the index is not visible to the optimizer, while visibility 1.0 means the index is fully visible. A value in the range between 0.0 and 1.0 means the index will be visible to the corresponding fraction of queries. #101812

• Row level TTL now supports DESC order primary key columns. #101869

• Added the ST_BdPolyFromText built-in which copies the behavior of the PostGIS function. Takes in only a multilinestring geometry and returns a polygon. It will return an error if anything other than a multilinestring is input, and will return an error if internally a multipolygon is created for some reason. NULL inputs also return NULL. #102708

• SHOW SCHEDULES now shows the schedule options with which the schedules were created. SHOW SCHEDULES FOR BACKUP additionally shows if the schedule is a full or incremental backup schedule. #102890

• You can no longer use PREPARE with EXPLAIN ANALYZE statements. Previously, this was allowed, but attempts to EXECUTE the prepared EXPLAIN ANALYZE statements would result in an error. #103259

• ttl_expiration_expression now allows stable operators and functions. This allows intervals to be directly added to TIMESTAMPTZ expressions. See https://www.postgresql.org/docs/15/xfunc-volatility.html. #102974

• CockroachDB now allows INSERT commands in UDF statement bodies. #102773

• CockroachDB now allows UPDATE and UPSERT commands in UDF statement bodies. #102773

• The READ COMMITTED isolation level is now supported. It can be used in the following ways:

  • When starting a transaction, use BEGIN ISOLATION LEVEL READ COMMITTED.
  • After starting a transaction, but before performing reads or writes, use SET TRANSACTION ISOLATION READ COMMITTED.
  • Configure it as the default isolation level using the default_transaction_isolation session variable. To see the isolation level of the currently running transaction, use either SHOW TRANSACTION ISOLATION LEVEL or SHOW transaction_isolation.

  #103482

• Added version gates which require all nodes in a given cluster to have a minimum binary version number, which in turn is required for creating forward indexes on JSON columns and for ordering JSON columns. #101932

• CockroachDB now allows DELETE commands in UDF statement bodies. #103531

• Added a new cluster setting sql.auth.public_schema_create_privilege.enabled which controls whether users receive CREATE privileges on the public schema or not. The setting applies at the time that the public schema is created, which happens whenever a database is created. The setting is true by default. #103598

• EXPLAIN (DDL) statements now have descriptor, index, column, constraint, and other ID values decorated with names when available. There is now also a new EXPLAIN (DDL, SHAPE) statement that provides information on costly operations planned by the declarative schema changer, like which index backfills and validations will get performed. #103930

• A new statistic KV pairs read is now exposed on EXPLAIN ANALYZE output in some cases (when this number is different from the KV rows read statistic or when the VERBOSE option is requested). This new statistic is also added to the telemetry sampled query events. #104079

• The KV rows read statistic in EXPLAIN ANALYZE output has been renamed to KV rows decoded to better reflect its meaning. #104079

• Table names are now allowed in SELECT lists inside view and UDF definitions. #104929

• SHOW JOB WITH EXECUTION DETAILS for a backup job will regenerate the DistSQL plan diagram with per-node and per-processor progress information. This will help users better understand the state of a running backup job. #103145

• The crdb_internal.node_transactions and crdb_internal.cluster_transactions tables now have columns for isolation_level, priority, and quality_of_service. #105009

• The SHOW RANGES command will now emit span statistics when the DETAILS option is specified. The statistics are included in a new column named span_stats, as a JSON object. The statistics are calculated for the identifier of each row. SHOW RANGES WITH DETAILS will compute span statistics for each range. SHOW RANGES WITH TABLES, DETAILS will compute span statistics for each table, and so on. The span_stats JSON object has the following keys:

  • approximate_disk_bytes
  • [key|val|sys|live|intent]_count
  • [key|val|sys|live|intent]_bytes

  approximate_disk_bytes is an approximation of the total on-disk size of the given object.

  key_count is the number of meta keys tracked under key_bytes. key_bytes is the number of bytes stored in all non-system point keys, including live, meta, old, and deleted keys. Only meta keys really account for the "full" key; value keys only for the timestamp suffix.

  val_count is the number of meta values tracked under val_bytes. val_bytes is the number of bytes in all non-system version values, including meta values.

  sys_count is the number of meta keys tracked under sys_bytes. sys_bytes is the number of bytes stored in system-local key-value pairs. This tracks the same quantity as (key_bytes + val_bytes), but for system-local metadata keys (which aren't counted in either key_bytes or val_bytes).

  live_count is the number of meta keys tracked under live_bytes. live_bytes is the number of bytes stored in keys and values which can in principle be read by means of a Scan or Get in the far future, including intents but not deletion tombstones (or their intents). Note that the size of the meta key-value pair (which could be explicit or implicit) is included in this. Only the meta key-value pair counts for the actual length of the encoded key (regular pairs only count the timestamp suffix).

  intent_count is the number of keys tracked under intent_bytes. It is equal to the number of meta keys in the system with a non-empty Transaction proto. intent_bytes is the number of bytes in intent key-value pairs (without their meta keys).

  #103128

• Introduced the pg_lsn data type, which is used to store the lsn associated with replication. #105031

• Users now can issue one ALTER TABLE statement with a combination of any number of ADD COLUMN, any number of DROP COLUMN, one ALTER PRIMARY KEY, and any number of ADD CONSTRAINT clauses. For example, with this PR, we now support statements like:

  CREATE TABLE t (i INT PRIMARY KEY, j INT NOT NULL, k INT NOT NULL);  ALTER TABLE t ADD COLUMN p INT DEFAULT 30, ALTER PRIMARY KEY USING COLUMNS (j), DROP COLUMN k, ADD CHECK (i > 0);
  

  #99526

• Added the ability to add numeric values to LSNs, or sub a decimal value from a LSN. #105326

• Implemented the pg_lsn - pg_lsn = decimal built-in function, which subtracts 2 LSNs to return a decimal. #105326

• Added limited support for scalar PL/pgSQL functions. Supported statements are variable declarations, variable assignments, IF statements, simple LOOP statements (with no conditions), EXIT and CONTINUE statements, and RETURN statements. #104755

• Implemented the spatial built-in ST_AsMVTGeom. #105530

• Pg_class's relreplident field was previously unpopulated. It is now populated with d for all tables (as each table has a primary key) and n otherwise. #106242

• Added the pg_sequence_last_value built-in function, which returns the last value generated by the sequence. #106445

• RESTORE can now be passed a WITH EXECUTION LOCALITY option similar to BACKUP, to restrict execution of the job to nodes with matching localities. #104439

• Added the REPLICATION user role option, which allows a user to use the streaming replication protocol. There is a corresponding REPLICATION system privilege. #106082

• A new view-only session variable, max_connections was added. This can be used with SHOW to view the maximum amount of non-superuser SQL connections allowed at a given time. #106952

• Added the nameconcatoid built-in function, which concatenates a name with an OID. #105944

• The pg_catalog.pg_language table is now populated with data about the languages used to define functions. #105944

• The information_schema.routines view is now populated with information about functions. #105944

• The information_schema.parameters table is now populated with information about function parameters. #105944

• Added support for the PLpgSQL RAISE statement, which allows sending notices to the client and raising errors. Currently the notice is only sent to the client. Support for logging notices will be added in a future release. #106351

• The public pseudo-role now receives the EXECUTE privilege by default for all user-defined functions that are created. This can be adjusted by using ALTER DEFAULT PRIVILEGES. #107317

• The crdb_interanal.node_statement_statistics table redacts the error message if the user has the VIEWACTIVITYREDACTED privilege, and does not redact the error message if the user has VIEWACTIVITY. If the user has both, VIEWACTIVITYREDACTED takes precedence and the last error is redacted. #107076

• The crdb_internal.cluster_locks table now has a isolation_level column indicating the isolation level. #107309

• In CommonSQLExecDetails, which is emitted as part of the SQL audit logs, SQL exec logs, and telemetry events, there is a new field: StmtPosInTxn. It represents the statement's index in the transaction, starting at 1. #107081

• cluster_logical_timestamp now returns an error when called at isolation levels lower than SERIALIZABLE. #107090

• EXPLAIN ANALYZE output now includes:

  • The isolation level of the statement's transaction.
  • The priority of the statement's transaction.
  • The quality of service level of the statement's transaction.

  #105857

• Added a new session variable, enable_implicit_fk_locking_for_serializable, which controls locking during foreign key checks under SERIALIZABLE isolation. With this set to true, foreign key checks of the referenced (parent) table, such as those performed during an INSERT or UPDATE of the referencing (child) table, will lock the referenced row using SELECT FOR SHARE locking. This is somewhat analogous to the existing enable_implicit_select_for_update variable but applies to the foreign key checks of a mutation statement instead of the initial row fetch. Under weaker isolation levels such as read committed, SELECT FOR SHARE locking will always be used to ensure the database maintains the foreign key constraint, regardless of the current setting of enable_implicit_fk_locking_for_serializable. #105857

• Add a new session variable, enable_durable_locking_for_serializable, which controls locking durability under SERIALIZABLE isolation. With this set to true, SELECT FOR UPDATE locks, SELECT FOR SHARED locks, and constraint check locks (e.g., locks acquired during foreign key checks if enable_implicit_fk_locking_for_serializable is set to true) will be guaranteed-durable under serializable isolation, meaning they will always be held to transaction commit. These locks are always guaranteed-durable under weaker isolation levels. By default, under serializable isolation these locks are best-effort rather than guaranteed-durable, meaning in some cases (e.g., leaseholder transfer, node loss, etc.) they could be released before the transaction commits. Serializable isolation does not rely on locking for correctness, only using it to improve performance under contention, so this default is a deliberate choice to avoid the performance overhead of lock replication. #107749

• The cluster setting server.cpu_profile.enabled has been removed. server.cpu_profile.cpu_usage_combined_threshold can enable and disable CPU profiling. #107717

• Added support for CONSTANT variable declarations in PLpgSQL routines. Any assignment to a variable declared with the CONSTANT keyword will raise a compile-time error. #107682

• Added a new syntax to SHOW DEFAULT PRIVILEGES, SHOW DEFAULT PRIVILEGES FOR GRANTEE <grantee>, that shows the default privileges that a grantee received. #107953

• The Statement diagnostics feature has been extended to support collecting a bundle for a particular plan. Namely, the existing fingerprint-based matching has been extended to also include plan-gist-based matching. Such bundles will miss a couple of things: plan.txt file as well as the tracing of the optimizer. At the moment, the feature is only exposed via an overload to the crdb_internal.request_statement_bundle built-in function. We now also support "anti-match": collecting a bundle for any plan other than the provided plan gist. #105477

• SHOW BACKUP's timestamp columns are now TIMESTAMPTZ, meaning they render in the session offset. #108290

• Attempting to drop a column when safe updates are enabled (sql_safe_updates = on) now additionally warns users that indexes referencing that column will be automatically dropped. #108047

• NOTICEs are now emitted for each index dropped by an ALTER TABLE ... DROP COLUMN ... statement. #108047

• SHOW JOBS now returns times (created, last_run, and so on) using the TIMESTAMPTZ column type instead of the TIMESTAMP type, meaning they are now rendered using the session offset. #108353

• Added a cluster setting sql.schema.force_declarative_statements to enable/disable DDL in the declarative schema changer. #107815

• Added the new built-in functions workload_index_recs() and workload_index_recs(TIMESTAMPTZ), which return workload level index recommendations (columns of string, each string represent an index recommendation) from statement level index recommendations (as candidates) in system.statement_statistics. If the TIMESTAMPTZ is given, it will only consider those candidates generated after that TIMESTAMPTZ value. #106525

• Added support for specifying PLpgSQL IF statements with ELSIF branches. #108211

• The admin API database details endpoint now returns authoritative range statistics. #108037

• Added the max_retries_for_read_committed session variable. It defaults to 10, and determines the number of times an individual statement in an explicit READ COMMITTED transaction will be retried if it encounters a retryable transaction error. #107044

• Added support for the execution of PLpgSQL functions with exception blocks. This allows a PLpgSQL function to catch and handle arbitrary errors it encounters during its execution. #107601

• Added the built-in functions bitmask_or, bitmask_and and bitmask_xor for variable-length input bitwise OR, AND, and XOR operations, respectively. #107863

• The oidvectortypes built-in has been implemented, which can format oidvector. #108467

• Added support for executing SQL statements directly within PLpgSQL routines. Note that this currently only applies to the subset of statements that can be executed within SQL UDFs, so CREATE TABLE is not supported, for example. INTO syntax is also supported. For example, SELECT * INTO a, b FROM xy;. #107920

• A SQL client can now request strict atomicity for mixed DDL/DML transactions with the new session variable strict_ddl_atomicity, which defaults to false. When this variable is set to true, CockroachDB will refuse to accept processing those specific DDL statements inside BEGIN...COMMIT for which it cannot guarantee atomic processing (other DDL statements are still allowed). Note that schema changes implicit in certain operations (e.g., IMPORT) are not protected via the new mechanism and can still fail with XXA00 errors. #42063

• Fixed an issue where the UI was missing query text and details on the SQL Activity Transactions page if there were more than 500 transactions or statements. The statement_activity table now includes all statements for a transaction that are in the transaction_activity table. #109424

• Added the VIEWSYSTEMTABLE system privilege. Users with this privilege have SELECT privileges for all tables in the system database. #109474

• The statement_activity and transaction_activity tables column execution_total_cluster_seconds is now accurate. The combinedstmts endpoint returns the correct value for the StmtsTotalRuntimeSecs and TxnsTotalRuntimeSecs properties. #109592

• The persistedsqlstats table maximum size check is now done once an hour instead of every 10 minutes. This reduces the risk of serialization errors on the statistics tables. #109696

• The deprecated session variable idle_in_session_timeout is now hidden from introspection. It was previously changed to idle_session_timeout. #109872

• The session variable ssl is now visible through introspection for better compatibility with PostgreSQL. #109872

• The session variable session_user is now invisible through introspection, in a way consistent with session_authorization and PostgreSQL. #109872

• There is now a CREATEROLE system privilege, which is analogous to the existing CREATEROLE role option, but can also be inherited by role membership. #109258

• Added the gen_random_bytes built-in function, which generates cryptographically secure random bytes. #110107

• The hash function used by hash-sharded indexes was changed to mod(fnv32(md5(crdb_internal.datums_to_bytes(columns))), bucket_count). Previously, it did not use md5. This change was made to enhance the uniformity of bucket distribution in cases when the bucket count is a power of 2, and the columns being sharded have numerical properties that make the fnv32 function return values with a non-uniformly distributed modulus. #109374

• New datetime built-ins (make_date, make_timestamp, and make_timestamptz) have been added, allowing for the creation of timestamps, timestamps with time zones, and dates. In addition, date_trunc now allows for a timestamp to be truncated in a specified timezone (to a specified precision). #108824

• There is now a CREATELOGIN system privilege, which is analogous to the existing CREATELOGIN role option, but can also be inherited by role membership. #110220

• There is now a CREATEDB system privilege, which is analogous to the existing CREATEDB role option, but can also be inherited by role membership. #110220

• There is now a CONTROLJOB system privilege, which is analogous to the existing CONTROLJOB role option, but can also be inherited by role membership. #110220

• The persistedsqlstats table maximum size check is now done once an hour instead of every 10 minutes. This reduces the risk of serialization errors on the statistics tables. #110173

• The new cluster setting sql.txn.read_committed_syntax.enabled, controls whether transactions run under READ COMMITTED or SERIALIZABLE isolation. It defaults to false. When set to true, the following statements will configure transactions to run under READ COMMITTED isolation:

  • BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED
  • SET TRANSACTION ISOLATION LEVEL READ COMMITTED
  • SET default_transaction_isolation = 'read committed'
  • SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ COMMITTED

  #110624

• The cluster setting sql.metrics.statement_details.gateway_node.enabled now defaults to false, to reduce the number of rows generated in SQL Statistics pages. #107788

• The default value for the ttl_job_cron table storage parameter is now @daily rather than @hourly. This parameter controls the default recurrence of the row-level TTL job. As part of this change, the output of the SHOW CREATE TABLE statements now include the ttl_cron_job parameter only if it is explicitly set. #110623

Operational changes

• Removed a timeseries metric that has not been reported for several versions. #100524
• Added two new metrics, range.snapshots.(send|recv)-queue-bytes, to track the total size of all snapshots waiting in the snapshot queue. #100942
• Exposed a new metric storage.compactions.duration, computed by the storage engine, that provides the cumulative time the storage engine has spent in compactions. This duration may exceed time elapsed, because of concurrent compactions, and may be useful in monitoring compaction concurrency. #103670
• Two new store metrics, range.snapshots.cross-region.sent-bytes and range.snapshots.cross-region.rcvd-bytes, were added to track the aggregate of snapshot bytes sent from and received at a store across different regions. Note that these metrics require the nodes' localities to include a “region” tier key. If a node lacks this key but is involved in cross-region batch activities, an error message will be logged. #104111

• Added new store metrics to track the aggregate of snapshot bytes sent from and received at a store across different zones.

  • range.snapshots.cross-zone.sent-bytes
  • range.snapshots.cross-zone.rcvd-bytes

  For accurate metrics, follow these recommendations: - Configure region and zone tier keys consistently across nodes. - Within a node locality, ensure unique region and zone tier keys. - Maintain consistent configuration of region and zone tiers across nodes.

  #104417

• Added new store metrics:

  • raft.rcvd.bytes
  • raft.sent.bytes
  • raft.rcvd.cross_region.bytes
  • raft.sent.cross_region.bytes
  • raft.rcvd.cross_zone.bytes
  • raft.sent.cross_zone.bytes

  #105122

• Added new DistSender metrics:

  • distsender.batch_requests.replica_addressed.bytes
  • distsender.batch_responses.replica_addressed.bytes
  • distsender.batch_requests.cross_region.bytes
  • distsender.batch_responses.cross_region.bytes
  • distsender.batch_requests.cross_zone.bytes
  • distsender.batch_responses.cross_zone.bytes.

  #103963

• Added new Node metrics:

  • batch_requests.bytes
  • batch_responses.bytes
  • batch_requests.cross_region.bytes
  • batch_responses.cross_region.bytes
  • batch_requests.cross_zone.bytes
  • batch_responses.cross_zone.bytes

  #104585

• Added new RPC metrics to help you to diagnose RPC connection issues:

  • grpc.connection.avg_round_trip_latency
  • rpc.connection.failures
  • rpc.connection.healthy
  • rpc.connection.healthy_nanos
  • rpc.connection.heartbeats
  • rpc.connection.unhealthy
  • rpc.connection.unhealthy_nanos

  #99191

• Added a new metric changefeed.lagging_ranges that shows the number of ranges which are behind in changefeeds. This metric can be used with the metrics_label changefeed option. Added a new changefeed option lagging_ranges_threshold, which is the amount of time a range needs to be behind to be considered lagging. By default this is 3 minutes. Added a new option lagging_ranges_polling_interval, which controls how often the lagging ranges calculation is done. This setting defaults to polling every 1 minute. Note that polling adds latency to the metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update for an additional minute afterwards. Also note that ranges undergoing an initial scan for longer than the threshold are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. However, as ranges complete the initial scan, the number of ranges will decrease. #109835

• A histogram metric raft.replication.latency was added. It tracks the time between evaluation and application of the command. This includes time spent in the quota pool, in replication (including re-proposals) as well as log application, but notably not sequencing latency (i.e., contention and latch acquisition). #106094

• The default Raft scheduler concurrency cap has been increased from 96 to 128 workers, scaling with 8 workers per CPU up to the cap. The scheduler concurrency can be controlled using the COCKROACH_SCHEDULER_CONCURRENCY environment variable. #105521

• The new cluster setting server.hot_ranges_request.node.timeout controls the maximum amount of time that a hot ranges request will spend waiting for a node to provide a response. It defaults to 5 minutes. To disable timeouts, set it to 0. #107796

• Two new cluster settings control whether intent resolution is subject to admission control: kv.intent_resolver.send_immediately.bypass_admission_control.enabled and kv.intent_resolver.batch.bypass_admission_control.enabled. #109932

• The new cluster setting admission.l0_min_size_per_sub_level reduces the probability of admission control throttling when there is a sequence of small memtable flushes or small files ingested into L0. #109332

• The new cluster setting kv.intent_resolver.batcher.in_flight_backpressure_limit.enabled controls whether an in-flight RPC limit is enforced on intent resolution RPCs. It defaults to false. #109899

• BACKUP now skips contacting the ranges for tables on which exclude_data_from_backup is set, and can thus succeed even if an excluded table is unavailable. #108627

• Span stats requests will return a partial result if the request encounters any errors. Errors that would have previously terminated the request are now included in the response. #108456

• The rangefeed closed timestamp interval controlled by kv.rangefeed.closed_timestamp_refresh_interval now defaults to 3 seconds. This affects how often rangefeeds emit resolved timestamps, and thus how often changefeeds can emit checkpoints. Previously, its default value of 0 would fall back to kv.closed_timestamp.side_transport_interval, which defaults to 200 milliseconds. Users who rely on the setting kv.closed_timestamp.side_transport_interval to control the rangefeed closed timestamp interval should make sure they either set kv.rangefeed.closed_timestamp_refresh_interval to 0 to retain the old behavior (preferably before upgrading), or to an appropriate value. #108667

• The default value of timeout for http-servers logging sinks has been changed from 0 (i.e., "no timeout") to 2s. This is reflected in the http-defaults section of the log configuration. Users still maintain the ability to override the timeout, or disable it by explicitly setting it to 0 (e.g. timeout: 0). #109264

• Changefeed metrics now include a changefeed.checkpoint_progress metric which is similar to changefeed.max_behind_nanos but supports metrics labels, as well as a changefeed.aggregator_progress metric which can track the progress of individual aggregators (the lowest timestamp for which all aggregators with the label have emitted all values they're responsible for). #108757

• Added support for Prometheus native histograms behind an environment variable flag. #104302

• Requests for database details or table details from the UI, or usages of [SHOW RANGES WITH DETAILS]/docs/v23.2/show-ranges.html are no longer subject to errors if the number of requested spans is too large. #109464

• The cockroach debug zip command now has an option to omit goroutine stack dumps. This impacts the creation of nodes/*/stacks.txt and nodes/*/stacks_with_labels.txt within debug ZIP bundles. Users can opt to exclude these goroutine stacks by using the --include-goroutine-stacks=false flag. Note that fetching stack traces for all goroutines is a "stop-the-world" operation, which can momentarily have negative impacts on SQL service latency. Note also that any periodic goroutine dumps previously taken on the node will still be included in nodes/*/goroutines/*.txt.gz, as these would have already been generated and don't require any stop-the-world operations. #110177

• New rangefeed metrics help to troubleshoot rangefeed restarts. The metric names have the format distsender.rangefeed.retry.{reason}. #109346

• Rangefeeds regularly attempt to push long-running transactions to a future timestamp in order to emit checkpoints. The interval at which this is attempted has been increased from 250 milliseconds to 1 seconds. This is now configurable via the environment variable COCKROACH_RANGEFEED_PUSH_TXNS_INTERVAL. #110332

Cluster virtualization

When cluster virtualization is enabled:

• A selection box displays in DB Console Metrics pages when you are connected to the system virtual cluster, and allows you to view metrics for a specific virtual cluster. #103308
• A "no data" empty graph state has been added when switching to a virtual cluster with no data. #103971
• A selection box displays on custom charts in the DB Console and allows you to select a specific virtual cluster. #103780
• The name of the virtual cluster, when known, is now reported in logging events. #108807
• When cockroach debug zip is run for a cluster with virtualization enabled, data about virtual clusters is now stored in a virtual subdirectory rather than a tenants subdirectory. #106117
• When cluster virtualization is enabled, the following closed timestamp side-transport settings can be set only from the system virtual cluster: kv.closed_timestamp.target_duration, kv.closed_timestamp.side_transport_interval, and kv.closed_timestamp.lead_for_global_reads_override. #108678

Command-line changes

• The CLI commands that output SQL data now support the JSON output format (--format=json), in addition to newline-delimited JSON (ND-JSON, --format=ndjson) that had been supported since v22.2. #102595
• cockroach debug zip now supports the command-line flag --format to select the format used to store SQL table data, in the same way as cockroach sql. In contrast to cockroach sql however, its default value is json (resulting in files named .json) and the default is not dependent on whether the terminal is interactive. #102607

• The SQL shell now supports argument quoting for client-side commands in a similar way to psql: inside single quotes, \ can escape characters and recognize octal/hexadecimal sequences; and inside double quotes characters are passed through. The quote characters themselves, when doubled, result in themselves as part of the string.

  For example, the following commands both result in a SQL prompt that says go "world":

  \set prompt1 'go "world"'
  \set prompt1 go' '"world"
  

  To add color to the prompt:

  \set prompt1 '\033[34mmydb>\033[m'
  

  These quoting rules are similar to PostgreSQL, but are different from the rules used by POSIX shells and of other programming languages like Python or Go. For example, octal and hex escape sequences support a variable number of digits, and double quoted strings preserve the surrounding quotes. When in doubt, refer to the PostgreSQL documentation.

  #104610

• The configuration for log output sinks now accepts a new format-options field. This can be used to customize the output of a given format. Each format accepts different options. One available option for the json output format is datetime-format.

  For example:

  sinks:
    fluent-groups:
      custom-json:
        format: json
        format-options: {datetime-format: rfc3339}
  

  This introduces a (new) field datetime in each output JSON event, with the format specified by the option. As of this writing, the following values are documented:

  • none: disable the creation of the datetime field. This is the default value.
  • iso8601 / rfc3339: format the time stamp like "2006-01-02T15:04:05.999999999Z".
  • rfc1123: format the time stamp like "Mon, 02 Jan 2006 15:04:05 +0000".

  Enabling the datetime field introduces CPU overhead and is not recommended. When using output to a log collector such as Fluent or Datadog, the log collector can be configured to transform the timestamp provided by CockroachDB without requiring participation from CockroachDB itself. When inspecting a log file containing JSON output produced by CockroachDB, the command cockroach debug merge-log can consume the JSON data and reformat it using the crdb-v2 format which also includes the date and time using the RFC3339 format. #104265

• The json log output format now recognizes the extra format option datetime-timezone which selects which timezone to use when formatting the datetime field. datetime-timezone must be combined with datetime-format because the default value for the latter option is none (i.e., datetime is not produced by default). For example:

  sinks:
    fluent-groups:
      custom-json:
        format: json
        format-options: {datetime-format: rfc3339, datetime-timezone: America/New_York}
  

  #104265

• The json log format now recognizes the format options tag-style and fluent-tag. The existing formats json-compact, json-fluent, json-fluent-compact have been redefined to become aliases for json with different defaults for the two new options. #104265

• The crdb-v1 log format now recognizes the format options show-counter and colors. The existing formats crdb-v1-tty, crdb-v1-count, crdb-v1-tty-count have been redefined to become aliases for crdb-v1 with different defaults for the two new options. #104265

• The crdb-v2 log format now recognizes the format option colors. The existing formats crdb-v2-tty has been redefined to become aliases for crdb-v2 with a different default for the new option. #104265

• The log output formats crdb-v1 and crdb-v2 now support the format option timezone. When specified, the corresponding time zone is used to produce the timestamp column. For example:

  file-defaults:
      format: crdb-v2
      format-options: {timezone: america/new_york}
  

  Example logging output:

  I230606 12:43:01.553407-040000 1 1@cli/start.go:575 ⋮ [n?] 4  soft memory limit of Go runtime is set to 35 GiB                        ^^^^^^^ indicates GMT-4 was used
  

  The timezone offset is also always included in the format if it is not zero (e.g., for non-UTC time zones). This is necessary to ensure that the times can be read back precisely. #104265

• The command cockroach debug merge-log was adapted to understand time zones in input files read with format crdb-v1 or crdb-v2. #104265

• When customizing the SQL interactive prompt, %M and %m now behave more like psql when connecting over a Unix datagram socket. #105137

• The default value of the --format parameter to cockroach debug zip is tsv, like other CLI commands that can extract SQL data. #107474

• The debug.zip now includes the crdb_internal.probe_range table with a limit of 100 rows to prevent the query from taking too long. #107720

• The default value for the --max-sql-memory parameter of the cockroach demo command has been increased from 128 MiB to 256 MiB. #103642

• The command \demo recommission has been removed from cockroach demo. It had been obsolete and non-functional ever since v20.2. #108566

• Added limited statement_statistics to the debug ZIP file. #108210

• The following user-visible cluster settings have been renamed. The previous name is still available for backward compatibility.

  Previous name	New name
  server.web_session_timeout	server.web_session.timeout
  kv.closed_timestamp.follower_reads_enabled	kv.closed_timestamp.follower_reads.enabled
  kv.range_split.by_load_enabled	kv.range_split.by_load.enabled
  changefeed.balance_range_distribution.enable	changefeed.balance_range_distribution.enabled
  changefeed.batch_reduction_retry_enabled	changefeed.batch_reduction_retry.enabled
  server.clock.forward_jump_check_enabled	server.clock.forward_jump_check.enabled
  server.oidc_authentication.autologin	server.oidc_authentication.autologin.enabled
  sql.metrics.statement_details.dump_to_logs	sql.metrics.statement_details.dump_to_logs.enabled
  sql.trace.log_statement_execute	sql.log.all_statements.enabled
  trace.debug.enable	trace.http_debug_endpoint.enabled

  #109074

• The following cluster settings have been renamed. The previous names are available for backward-compatibility.

  Previous name	New name
  spanconfig.tenant_coalesce_adjacent.enabled	spanconfig.range_coalescing.application.enabled
  spanconfig.storage_coalesce_adjacent.enabled	spanconfig.range_coalescing.system.enabled

  #109077

• The new cockroach gen metric-list command generates metadata that describes the various metrics collected by an idle server. The list does not include dynamic metric names whose names are generated based on the workload. #109042