How to Migrate an on-premise CRDB to Redis Cloud CRDB?

Redis Cloud
Redis Enterprise Software
crdb

Last updated 18, Apr 2024

Question

What are the options for migrating a CRDB database from on-premise to Redis Cloud with minimal downtime?

Answer

RIOT-REDIS is a Java-based tool that you can use to migrate data from different sources (on-prem OSS Redis, on-prem Redis Enterprise Software, ElastiCache). To install this tool, you will have to create an instance that runs next to your source database clusters and ensure that this instance has access to both the source database and the target database.

NOTE: Another alternative to RIOT-Redis is the import functionality. This option almost always assumes a longer downtime and several manual tasks regarding merging multiple RDB files into one. Since the CRDB database automatically syncs between the clusters, we will select one database endpoint for the source database and one for the target database. (NOTE: The rest of the regional databases will catch up automatically, and it is sufficient to bring the data to only one target database).

To install RIOT, you must follow the official RIOT-REDIS documentation.

Command to initiate the migration

Depending on the path where you installed RIOT-Redis, the command to initiate the migration is given below. You must execute this command on your RIOT-Redis instance.

./riot-redis --info -h <DB_SOURCE_ENDPOINT> -p <DB_SOURCE_PORT> --pass <DB_SOURCE_PASSWORD> replicate-ds -h <DB_TARGET_ENDPOINT> -p <DB_TARGET_ENDPOINT> --pass <DB_TARGET_PASSWORD> --mode live

Here is the breakdown of the command.

  • ./riot-redis is the path where I installed RIOT-Redis
  • --info provides visibility into how to program executes once you trigger the command
  • -h <DB_SOURCE_ENDPOINT> -p <DB_SOURCE_PORT> is the source database endpoint and the port number
  • --pass <DB_SOURCE_PASSWORD> is the password of the source database
  • replicate-ds is a method of running replication; if you are using CRDB, this is the only way to do it (Read more here)
  • -h <DB_TARGET_ENDPOINT> -p <DB_TARGET_ENDPOINT> is the target database (Redis Cloud) together with the port number
  • --pass <DB_TARGET_PASSWORD> is the password of the target database
  • --mode live makes sure that once you run the command, the initial dataset will be replicated plus the incoming changes (the list of all the available modes is here)

Known limitations and prerequisites

CRDB databases do not support the RESTORE command RIOT-Redis uses to copy the initial dataset. Because of that, the replicate mechanism in RIOT-Redis will fail. The only replication mechanism that supports CRDB is replicate-ds. The difference here is that Redis-RIOT uses the read/write command for every key, depending on the key type, rather than the RESTORE command. (For strings, it would use GET foo bar and SET foo bar rather than RESTORE). To use the live --mode replication, you must execute the following command on both source and target clusters. CONFIG SET notify-keyspace-events KA This is because the live method uses key scan and keyspace notifications to track the changes in the keyspace.

References

RIOT-REDIS