Deploy a pipeline

Learn how to deploy an RDI pipeline

The sections below explain how to deploy a pipeline after you have created the required configuration.

Set secrets

Before you deploy your pipeline, you must set the authentication secrets for the source and target databases. Every secret belongs to one database: a source, identified by its name in config.yaml, or the target. You name that database with the --db option of the redis-di set-secret command.

The table below lists the available secret keys. The username and password are required, while the other keys are only relevant for TLS/mTLS connections.

You can reference a secret in config.yaml using an environment variable that is derived from the secret key and the database name. The variable name consists of the database name in uppercase (with each dash replaced by an underscore), followed by _DB_, followed by the key. For example, if you set PASSWORD with --db mysql the corresponding environment variable is MYSQL_DB_PASSWORD, which the source references as ${MYSQL_DB_PASSWORD}. If you set PASSWORD with --db target, the environment variable is TARGET_DB_PASSWORD. The sample config.yaml file shows these references in use, and Multiple sources in one pipeline covers the source naming rules.

openssl pkcs8 -topk8 -inform PEM -outform DER \
    -in /path/to/myclient.pem \
    -out /path/to/myclient.pk8 -nocrypt

Set secrets with the CLI

Use redis-di set-secret to set secrets for any installation type (VM, Kubernetes, or Redis Cloud).

The command lines for a source named mysql are shown below. If your pipeline has multiple sources, you should run the command once for each source, using the appropriate database name.

# For username and password
redis-di set-secret USERNAME --db mysql yourUsername
redis-di set-secret PASSWORD --db mysql yourPassword

# With source TLS, in addition to the above
redis-di set-secret CACERT --db mysql /path/to/myca.crt

# With source mTLS, in addition to the above
redis-di set-secret CERT --db mysql /path/to/myclient.crt
redis-di set-secret KEY --db mysql /path/to/myclient.key
# Use this only if the private key is password-protected
redis-di set-secret KEY_PASSWORD --db mysql yourKeyPassword

The corresponding command lines for target secrets are:

# For username and password
redis-di set-secret USERNAME --db target yourUsername
redis-di set-secret PASSWORD --db target yourPassword

# With target TLS, in addition to the above
redis-di set-secret CACERT --db target /path/to/myca.crt

# With target mTLS, in addition to the above
redis-di set-secret CERT --db target /path/to/myclient.crt
redis-di set-secret KEY --db target /path/to/myclient.key
# Use this only if the private key is password-protected
redis-di set-secret KEY_PASSWORD --db target yourKeyPassword

By default, set-secret waits for the pipeline to apply the change before returning. When you set several secrets at once, set all but the last one with --wait=false to avoid a timeout while the pipeline is only partially updated. See Wait for changes to complete below for details.

Manage secrets with the CLI

Along with set-secret, the CLI has commands to list, inspect, and delete secrets. Because the API never returns secret values, these commands show only the secret keys and whether they are set, not the stored values.

# List all the secrets of a pipeline, with the database each one belongs to
redis-di list-secrets

# List only the secrets of one database
redis-di list-secrets --db mysql

# Show a single secret and whether it is set
redis-di describe-secret PASSWORD --db mysql

# Delete a secret (prompts for confirmation unless you add --force)
redis-di delete-secret CACERT --db mysql

See the reference pages for list-secrets, get-secret, describe-secret, and delete-secret for the full list of options.

Set secrets for K8s/Helm deployment using Kubectl command

For a Kubernetes/Helm deployment, you can also use kubectl create secret generic to set secrets instead of the CLI. The general pattern of the commands is:

kubectl create secret generic <DB> \
--namespace=rdi \
--from-literal=<SECRET-NAME>=<SECRET-VALUE>

Where <DB> is <source>-db for the secrets of a source, or target-db for target secrets. The examples below use a source named mysql, so its secret is mysql-db.

If you use TLS or mTLS for either the source or target databases, you also need to create the <source>-db-ssl and/or target-db-ssl K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is:

kubectl create secret generic <DB>-ssl \
--namespace=rdi \
--from-file=<FILE-NAME>=<FILE-PATH>

The specific command lines for source secrets are as follows:

# Without source TLS
# Create or update mysql-db secret
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source TLS
# Create or update mysql-db secret
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update mysql-db-ssl secret
kubectl create secret generic mysql-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source mTLS
# Create or update mysql-db secret. Include the MYSQL_DB_KEY_PASSWORD line
# only if the private key is password-protected.
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \
--from-literal=MYSQL_DB_CERT=/etc/certificates/mysql_db/client.crt \
--from-literal=MYSQL_DB_KEY=/etc/certificates/mysql_db/client.key \
--from-literal=MYSQL_DB_KEY_PASSWORD=yourKeyPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update mysql-db-ssl secret
kubectl create secret generic mysql-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -

The corresponding command lines for target secrets are:

# Without target TLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target TLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target mTLS
# Create or update target-db secret. Include the TARGET_DB_KEY_PASSWORD line
# only if the private key is password-protected.
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \
--from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \
--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -

Note that the certificate paths contained in the CACERT, CERT, and KEY secrets are internal to RDI, so you must use the values shown in the example above. Each source has its own certificate directory, named after the source (for example, /etc/certificates/mysql_db/ for a source named mysql). You should only change the certificate paths when you create the <source>-db-ssl and target-db-ssl secrets.

You must also label any secrets that you create directly with kubectl so that the RDI operator discovers them as pipeline secrets. Give each secret the following labels, where the app.kubernetes.io/instance label corresponds to the pipeline name (the name is just default for the default pipeline):

Apply the labels to each secret with kubectl label:

kubectl label secret mysql-db --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi
kubectl label secret target-db --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

# With source TLS or mTLS
kubectl label secret mysql-db-ssl --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

# With target TLS or mTLS
kubectl label secret target-db-ssl --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

Deploy a pipeline

When you have created your configuration, including the jobs, you are ready to deploy. Use the redis-di deploy command to deploy a pipeline:

redis-di deploy --dir <path to pipeline folder>

RDI first validates the configuration and then deploys it if it is correct. You can control the validation and what happens after deployment with the following options:

• --dry-run: Validate the configuration without deploying it. Off by default.
• --validate-tables: Validate the configuration against the source and target databases, for example that the tables it references exist. On by default; pass --validate-tables=false to skip this check, which is useful when the databases are not reachable at deploy time.
• --validate-cdc: Additionally validate that the source database is correctly configured for change data capture (CDC). Off by default; enable it with --validate-cdc.
• --start: Start the pipeline as soon as it is deployed. On by default; pass --start=false to deploy the pipeline without starting it, then start it later with redis-di start.

See the redis-di deploy reference page for the full list of options.

You can also use Redis Insight to configure and deploy pipelines for both VM and K8s installations.

Display the pipeline status

Once a pipeline is deployed, use the redis-di describe command (also available as redis-di status) to display its status. This combines the pipeline configuration with its runtime status, showing its overall state, its sources and targets, its jobs and components, and its per-stream statistics and performance metrics.

redis-di describe

To watch the status update live, pair the command with watch:

watch -n 1 redis-di describe

For a shorter overview, redis-di list prints a one-line summary of the pipeline, and redis-di get does the same for a single pipeline. See the redis-di describe reference page for details.

Start and stop a pipeline

Use redis-di stop to pause a running pipeline and redis-di start to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its configuration, so you can start it again later from where it left off.

redis-di stop
redis-di start

To act on a single source instead of the whole pipeline, add --source:

redis-di stop --source mysql
redis-di start --source mysql

Note that a source can only run if its parent pipeline is running. See Multiple sources in one pipeline for more information.

Reset a pipeline

Use redis-di reset to return a pipeline to initial full-sync mode. This reloads a fresh snapshot of the source data and then resumes change data capture (CDC), which is useful when the source and target have drifted out of sync.

redis-di reset

Add --source to reset a single source. The other sources keep their data, but the whole pipeline stops while the reset runs and starts again afterwards:

redis-di reset --source mysql

Clear a pipeline

To stop a pipeline and discard its configuration, deploy an empty configuration with the --empty option of redis-di deploy:

redis-di deploy --empty

RDI removes the pipeline's data plane components and deletes its data from the RDI database, including the change data streams, offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters of every source. The pipeline itself remains, with an empty configuration, so you can deploy a new configuration to it at any time.

The pipeline secrets are not affected, so remove them yourself with redis-di delete-secret if you no longer need them. The records the pipeline wrote to the target database are not deleted either.

Wait for changes to complete

The commands that change a pipeline's state, namely deploy, start, stop, reset, set-secret, and delete-secret, do not return as soon as the API accepts the request. By default, they wait for the pipeline to finish transitioning to the expected state, polling its status until it succeeds, reaches an error, or the --timeout (2 minutes by default) elapses. This is usually what you want: the command reflects the real outcome, so a script can rely on the change having taken effect and can fail fast if it did not.

In some cases, though, a pipeline needs several changes before it can transition to a healthy state, and waiting after each individual change would time out. The clearest example is rotating both the username and the password of a database: if you set only the username with the default --wait=true, the pipeline tries to reconnect with the new username and the old password, fails, and the command times out after two minutes with the pipeline in a broken state.

To avoid this, set all the related secrets, or at least all of them except the last, with --wait=false, so the pipeline applies them together and only the final command waits for it to become healthy:

redis-di set-secret USERNAME --db mysql newUsername --wait=false
redis-di set-secret PASSWORD --db mysql newPassword

The same applies to any set of changes that are only valid together.