MOLT Releases

On this page

This page has details about each release of the following MOLT (Migrate Off Legacy Technology) tools:

• Fetch
• Verify

Cockroach Labs recommends using the latest available version of each tool. See Installation.

Installation

To download the latest MOLT Fetch/Verify binary:

For previous binaries, refer to the MOLT version manifest.

Docker image

Docker multi-platform images containing both the AMD and ARM binaries are available. To pull the latest image:

docker pull cockroachdb/molt

To pull a specific version (e.g., 1.1.3):

docker pull cockroachdb/molt:1.1.3

October 29, 2024

MOLT Fetch/Verify 1.2.0 is available.

• Added failback mode to MOLT Fetch, which allows the user to replicate changes on CockroachDB back to the initial source database. Failback is supported for MySQL and PostgreSQL databases.
• The --pprof-list-addr flag, which specifies the address of the pprof endpoint, is now configurable. The default value is '127.0.0.1:3031'.
• Fetch modes involving replication now state that MySQL 8.0 and later are supported for replication between MySQL and CockroachDB.
• Partitioned tables can now be moved to CockroachDB using IMPORT INTO.
• Improved logging for the Fetch schema check phases under the trace logging level, which is set with --logging trace.
• Added a sample Grafana dashboard for monitoring MOLT tasks.
• Fetch now logs the name of the staging database in the target CockroachDB cluster used to store metadata for replication modes.
• String primary keys that use C collations on PostgreSQL can now be compared to the default en_US.utf8 on CockroachDB.
• MOLT is now distributed under the Cockroach Labs Product License Agreement, which is bundled with the binary.

August 26, 2024

MOLT Fetch/Verify 1.1.7 is available.

• When a Fetch transformation rule is used to rename a table or map partitioned tables, a script in the format partitionTableScript.{timestamp}.ts is now automatically generated to ensure that replication works properly if enabled.

August 19, 2024

MOLT Fetch/Verify 1.1.6 is available.

• Fixed a Fetch bug where --table-exclusion-filter was ignored when --table-filter and --schema-filter were set to the default ('.*').

August 15, 2024

MOLT Fetch/Verify 1.1.5 is available.

• Deprecated the --ongoing-replication flag in favor of --mode data-load-and-replication, using the new --mode flag. Users should replace all instances of --ongoing-replication with --mode data-load-and-replication.
• Fetch can now be run in an export-only mode by specifying --mode export-only. This will export all the data in csv or csv.gz format to the specified cloud or local store.
• Fetch can now be run in an import-only mode by specifying --mode import-only. This will load all data in the specified cloud or local store into the target CockroachDB database, effectively skipping the export data phase.
• Strings for the --mode flag are now word-separated by hyphens instead of underscores. For example, replication-only instead of replication_only.

August 8, 2024

MOLT Fetch/Verify 1.1.4 is available.

• Added a replication-only mode for Fetch that allows the user to run ongoing replication without schema creation or initial data load. This requires users to set --mode replication_only and --replicator-flags to specify the defaultGTIDSet (MySQL) or slotName (PostgreSQL).
• Partitioned tables can now be mapped to renamed tables on the target database, using the Fetch transformations framework.
• Added a new --metrics-scrape-interval flag to allow users to specify their Prometheus scrape interval and apply a sleep at the end to allow for the final metrics to be scraped.
• Previously, there was a mismatch between the errors logged in log lines and those recorded in the exceptions table when an IMPORT INTO or COPY FROM operation failed due to a non-PostgreSQL error. Now, all errors will lead to an exceptions table entry that allows the user to continue progress from a certain table's file.
• Fixed a bug that will allow Fetch to properly determine a GTID if there are multiple source_uuids.

July 31, 2024

MOLT Fetch/Verify 1.1.3 is available.

• 'infinity'::timestamp values can now be moved with Fetch.
• Fixed an issue where connections were not being closed immediately after sharding was completed. This could lead to errors if the maximum number of connections was set to a low value.
• Fetch users can now exclude specific tables from migration using the --table-exclusion-filter flag.

July 18, 2024

MOLT Fetch/Verify 1.1.2 is available.

• Fetch users can now specify columns to exclude from table migrations in order to migrate a subset of their data. This is supported in the schema creation, export, import, and direct copy phases.
• Fetch now automatically maps a partitioned table from a PostgreSQL source to the target CockroachDB schema.
• Fetch now supports column exclusions and computed column mappings via a new transformations framework.
• The new Fetch --transformations-file flag specifies a JSON file for schema/table/column transformations, which has validation utilities built in.

July 10, 2024

MOLT Fetch/Verify 1.1.1 is available.

• Fixed a bug that led to incorrect list continuation file behavior if a trailing slash was provided in --bucket-path.
• Fixed a bug with extracting the filename from a failed import URL. Previously, an older filename was being used, which could result in duplicated data. Now, the filename that is used in import matches what is stored in the exceptions log table.
• Added a --use-implicit-auth flag that determines whether implicit authentication is used for cloud storage import URIs.

July 8, 2024

MOLT Fetch/Verify 1.1.0 is available.

• Fixed a bug where integer values for JSONB/JSON types were not correctly handled when migrating from a PostgreSQL or CockroachDB source.
• Optimized the logic for clearing continuation tokens. Now, the clearing option is only presented in the relevant modes and when there are tokens to clear.
• CockroachDB sources now exclusively accept connections from admin or root users.
• The statement timeout can now be adjusted during the export phase. This guards against hanging queries.

May 31, 2024

MOLT Fetch/Verify 1.0.0 is available.

• Renamed the --table-splits flag to --concurrency-per-table, which is more descriptive.
• Increased the default value of --import-batch-size to 1000. This leads to better performance on the target post-migration. Each individual import job will take longer, since more data is now imported in each batch, but the sum total of all jobs should take the same (or less) time.

May 29, 2024

MOLT Fetch/Verify 0.3.0 is available.

• Added an --import-batch-size flag, which configures the number of files to be imported in each IMPORT job.
• In some cases on the previous version, binaries would not work due to how molt was being built. Updated the build method to use static linking, which creates binaries that should be more portable.
• VARBIT <> BOOL conversion is now allowed for Fetch and Verify. The bit array is first converted to UINT64. A resulting 1 or 0 is converted to true or false accordingly. If the UINT64 is another value, an error is emitted.

May 20, 2024

MOLT Fetch/Verify 0.2.1 is available.

• MOLT tools now enforce secure connections to databases as a default. The --allow-tls-mode-disable flag allows users to override that behavior if secure access is not possible.
• When using MySQL as a source, --table-concurrency and --export-concurrency are strictly set to 1.
• Fixed a bug involving history retention for DECIMAL values.

May 3, 2024

MOLT Fetch/Verify 0.2.0 is available.

• Fetch now supports CockroachDB multi-region tables.
• Fetch now supports continuous replication for PostgreSQL and MySQL source databases via the --ongoing-replication flag. When Fetch finishes the initial data load phase, it will start the replicator process as a subprocess, which runs indefinitely until the user ends the process with a SIGTERM (ctrl-c).
• Replicator flags for (PostgreSQL and MySQL) are now supported, allowing users to further configure the --ongoing-replication mode for their use case.
• Added the --type-map-file flag, which enables custom type mapping for schema creation.
• Fixed a bug where primary key positions could be missed when creating a schema with multiple primary keys.
• Added a default mode for MySQL sources that ensures consistency and does not leverage parallelism. New text is displayed that alerts the user and links to documentation in cases where fetching from MySQL might not be consistent.
• Logging for continuation tokens is now omitted when data export does not successfully complete.

April 10, 2024

MOLT Fetch/Verify 0.1.4 is available.

• Fixed an inconsistency where inserting 'null' as a string into JSON/JSONB columns did not align with PostgreSQL's case-sensitive null handling.

March 22, 2024

MOLT Fetch/Verify 0.1.2 is available.

• Fixed a bug where running Verify with a PostgreSQL or CockroachDB source would cause an error when there are dropped columns on the table.

March 13, 2024

MOLT Fetch/Verify 0.1.1 is available.

• An error is now displayed if the original column is a PostgreSQL nested array.
• Added sharded data export functionality for faster performance.
• Added support for the --table-handling 'drop-on-target-and-recreate' flag for MySQL sources, enabling automatic schema recreation.
• Added support for ENUM data types in schema creation.
• Deprecated the --s3-bucket and --gcp-bucket flags in favor of a single --bucket-path flag.

February 22, 2024

MOLT Fetch/Verify 0.1.0 is available.

• Added implicit authentication for GCP sources.
• Resolved a CSV write deadlock issue for large file exports.
• Added new dashboard metrics: molt_verify_num_tables (categorizes tables as verified, missing, or extraneous) and molt_verify_duration_ms (how long a Verify run took to complete, in milliseconds).
• Previously, if a comparison between heterogeneous column types occurred, Verify would panic and stop execution. Now, Verify no longer panics, but records the error and reports it to logging, and also marks the row as having mismatching columns. The error message is logged at most once per shard to avoid log spewing. Additionally, introduced the concept of conditional success, which means that all comparable row columns were compared and succeeded with the same value. Upon conditional success, the user is instructed to check all warnings and verify the data.
• Marked the --live and --continuous flags as dependencies for other sub-configuration flags. If a flag is set without its dependency, MOLT will indicate which dependency flag is not correctly set.
• Added the RFC for MOLT Fetch, which explains the architectural decisions and considerations of how Fetch was built. The RFC also outlines limitiations of the current system and further improvements around usability, performance, and integration with replication tools.
• Added logging annotations for data-level errors and information that surfaced during the Verify process, as well as summary status logs about the overall functioning of the job (e.g., 10000 truth rows seen, 5000 mismatches, etc.). These can be filtered so that exact tables and rows can be triaged in one place.
• Changed the output for the running summary statistics so that they are now more easily indexable by log processing tools. Previously, it was more human-readable, but fields were not easily indexable out of the box. Now, there is a balance between readability and downstream processing.
• Added tagging for summary and data logs so that users can filter the logs in a variety of log processing tools, or grep in the command line. This solves the pain point of all the logs going to the same location without any differentiation. Now, it is possible to view types of logs one at a time. Summary logs report aggregated results on either a task or step in a task (e.g., task took 10 minutes; 500 tables processed overall). Data logs report on individual or row-level operations and quantitative markers (e.g., 1000 rows completed so far; failed to verify item with PK 1000)
• Marked the --s3-bucket, --gcp-bucket, and --local-path flags as mutually exclusive because they all represent different intermediary stores.
• Adjusted settings for the number of rows and the size of data to be flushed so that only one rule is applied at any given time.
• Fixed a bug where outputs containing special characters were not formatted correctly.
• When running the import query for Fetch, the import statement is now printed if the logger is in debug mode.
• Updated the MOLT README with an OS + Docker-based edge case with Fetch.
• Logging for Fetch and Verify now defaults to structured JSON logging, which contains at least the message, level, and formatted time string. The --use-console-writer flag enables prettier console writing that has colored and human-readable log output. Structured logging is the default because it is easier to parse with log ingestion tools and is more efficient for the application to output.
• Task/data logs can now be written to specified log files using --log-file. If the flag is left empty, the MOLT tool will only write to stdout, which is more efficient.

Was this helpful?