Deploy a pipeline

Learn how to deploy an RDI pipeline

The sections below explain how to deploy a pipeline after you have created the required configuration.

Set secrets

Before you deploy your pipeline, you must set the authentication secrets for the source and target databases. Every secret belongs to one database: a source, identified by its name in config.yaml, or the target. You name that database with the --db option of the redis-di set-secret command. Every secret also belongs to one pipeline. The secret commands manage the secrets of the default pipeline unless you name another pipeline with the -p or --pipeline option. See Set secrets for each pipeline.

The table below lists the available secret keys. The username and password are required, while the other keys are only relevant for TLS/mTLS connections.

RDI also accepts a generic secret key for another connection parameter. The key can contain up to 63 uppercase letters, digits, and underscores, must start with a letter, and must use the --db option. The 63-character limit applies to the key you pass to redis-di set-secret, excluding the generated database prefix. For example, set a source-specific port with redis-di set-secret PORT --db mysql 3306 and reference it as ${MYSQL_DB_PORT}.

A generic key is a simple value by default. To store it as a file instead, such as an additional CA certificate, pass --type file. RDI then mounts the file, and the reference resolves to the path of the mounted file. For example:

redis-di set-secret EXTRA_CA --db mysql --type file --file /path/to/extra-ca.crt

Reference this secret as ${MYSQL_DB_EXTRA_CA} in config.yaml. It resolves to the mounted file's path.

The --file option only tells the command to read the value from a local file. It works with any key, and it does not change the type of the secret.

You can reference a secret in config.yaml using an environment variable that is derived from the secret key and the database name. The variable name consists of the database name in uppercase (with each dash replaced by an underscore), followed by _DB_, followed by the key. For example, if you set PASSWORD with --db mysql the corresponding environment variable is MYSQL_DB_PASSWORD, which the source references as ${MYSQL_DB_PASSWORD}. If you set PASSWORD with --db target, the environment variable is TARGET_DB_PASSWORD. The sample config.yaml file shows these references in use, and Multiple sources in one pipeline covers the source naming rules.

openssl pkcs8 -topk8 -inform PEM -outform DER \
    -in /path/to/myclient.pem \
    -out /path/to/myclient.pk8 -nocrypt

Set secrets with the CLI

Use redis-di set-secret to set secrets for any installation type (VM, Kubernetes, or Redis Cloud).

The command lines for a source named mysql are shown below. If your pipeline has multiple sources, you should run the command once for each source, using the appropriate database name.

# For username and password
redis-di set-secret USERNAME --db mysql yourUsername
redis-di set-secret PASSWORD --db mysql yourPassword

# With source TLS, in addition to the above
redis-di set-secret CACERT --db mysql /path/to/myca.crt

# With source mTLS, in addition to the above
redis-di set-secret CERT --db mysql /path/to/myclient.crt
redis-di set-secret KEY --db mysql /path/to/myclient.key
# Use this only if the private key is password-protected
redis-di set-secret KEY_PASSWORD --db mysql yourKeyPassword

The corresponding command lines for target secrets are:

# For username and password
redis-di set-secret USERNAME --db target yourUsername
redis-di set-secret PASSWORD --db target yourPassword

# With target TLS, in addition to the above
redis-di set-secret CACERT --db target /path/to/myca.crt

# With target mTLS, in addition to the above
redis-di set-secret CERT --db target /path/to/myclient.crt
redis-di set-secret KEY --db target /path/to/myclient.key
# Use this only if the private key is password-protected
redis-di set-secret KEY_PASSWORD --db target yourKeyPassword

By default, set-secret waits for the pipeline to apply the change before returning. When you set several secrets at once, set all but the last one with --wait=false to avoid a timeout while the pipeline is only partially updated. See Wait for changes to complete below for details.

Manage secrets with the CLI

Along with set-secret, the CLI has commands to list, inspect, and delete secrets. These commands show the key, database, type, and configuration reference of each secret, but not its value, because the API never returns secret values. The configuration reference is the placeholder to use in config.yaml.

# List all the secrets of a pipeline, with the database each one belongs to
redis-di list-secrets

# List only the secrets of one database
redis-di list-secrets --db mysql

# Show a single secret and its configuration reference
redis-di describe-secret PASSWORD --db mysql

# Delete a secret (prompts for confirmation unless you add --force)
redis-di delete-secret CACERT --db mysql

See the reference pages for list-secrets, get-secret, describe-secret, and delete-secret for the full list of options.

Set secrets for K8s/Helm deployment using Kubectl command

For a Kubernetes/Helm deployment, you can also use kubectl create secret generic to set secrets instead of the CLI. The general pattern of the commands is:

kubectl create secret generic <DB> \
--namespace=rdi \
--from-literal=<SECRET-NAME>=<SECRET-VALUE>

Where <DB> is <source>-db for the secrets of a source, or target-db for target secrets. The examples below use a source named mysql, so its secret is mysql-db. For a pipeline other than default, prefix each secret name with the pipeline name and two dashes, for example orders--mysql-db and orders--target-db for a pipeline named orders.

If you use TLS or mTLS for either the source or target databases, you also need to create the <source>-db-ssl and/or target-db-ssl K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is:

kubectl create secret generic <DB>-ssl \
--namespace=rdi \
--from-file=<FILE-NAME>=<FILE-PATH>

The specific command lines for source secrets are as follows:

# Without source TLS
# Create or update mysql-db secret
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source TLS
# Create or update mysql-db secret
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update mysql-db-ssl secret
kubectl create secret generic mysql-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source mTLS
# Create or update mysql-db secret. Include the MYSQL_DB_KEY_PASSWORD line
# only if the private key is password-protected.
kubectl create secret generic mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \
--from-literal=MYSQL_DB_CERT=/etc/certificates/mysql_db/client.crt \
--from-literal=MYSQL_DB_KEY=/etc/certificates/mysql_db/client.key \
--from-literal=MYSQL_DB_KEY_PASSWORD=yourKeyPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update mysql-db-ssl secret
kubectl create secret generic mysql-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -

The corresponding command lines for target secrets are:

# Without target TLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target TLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target mTLS
# Create or update target-db secret. Include the TARGET_DB_KEY_PASSWORD line
# only if the private key is password-protected.
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \
--from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \
--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -

Note that the certificate paths contained in the CACERT, CERT, and KEY secrets are internal to RDI, so you must use the values shown in the example above. Each source has its own certificate directory, named after the source (for example, /etc/certificates/mysql_db/ for a source named mysql). You should only change the certificate paths when you create the <source>-db-ssl and target-db-ssl secrets.

You must also label any secrets that you create directly with kubectl so that the RDI operator discovers them as pipeline secrets. Give each secret the following labels, where the app.kubernetes.io/instance label corresponds to the pipeline name (the name is just default for the default pipeline):

Apply the labels to each secret with kubectl label:

kubectl label secret mysql-db --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi
kubectl label secret target-db --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

# With source TLS or mTLS
kubectl label secret mysql-db-ssl --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

# With target TLS or mTLS
kubectl label secret target-db-ssl --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=default \
  product=rdi

Deploy a pipeline

When you have created your configuration, including the jobs, you are ready to deploy. A new RDI installation has no pipeline, so create the pipeline with an empty configuration first, and then deploy your configuration to it with the redis-di deploy command:

redis-di deploy --empty
redis-di deploy --dir <path to pipeline folder>

To validate your configuration against the source and target databases, create an empty pipeline first, then deploy your configuration. Running redis-di deploy --dir before the pipeline exists creates the pipeline, but doesn't run these database checks.

The pipeline name is optional and defaults to default. To deploy another pipeline, pass its name to both commands:

redis-di deploy orders --empty
redis-di deploy orders --dir <path to pipeline folder>

See Multiple pipelines in one RDI installation for the pipeline naming rules.

When you deploy a configuration to an existing pipeline, RDI first validates the configuration and then deploys it if it is correct. You can control the validation and what happens after deployment with the following options:

• --dry-run: Validate the configuration without deploying it. Off by default.
• --validate-tables: Validate the configuration against the source and target databases, for example that the tables it references exist. On by default; pass --validate-tables=false to skip this check, which is useful when the databases are not reachable at deploy time.
• --validate-cdc: Additionally validate that the source database is correctly configured for change data capture (CDC). Off by default; enable it with --validate-cdc.
• --start: Start the pipeline as soon as it is deployed. On by default; pass --start=false to deploy the pipeline without starting it, then start it later with redis-di start.

See the redis-di deploy reference page for the full list of options.

You can also use Redis Insight to configure and deploy pipelines for both VM and K8s installations.

Export a pipeline configuration

Use redis-di export to write the configuration of a deployed pipeline to a directory, as config.yaml and jobs/*.yaml. You can then edit the files and deploy them again, or deploy them to another pipeline or RDI instance:

redis-di export --dir <path to pipeline folder>
redis-di deploy --dir <path to pipeline folder>

The export contains secret references but no secret values, so set the secrets of the pipeline first when you deploy the export to another pipeline or RDI instance.

Display the pipeline status

Once a pipeline is deployed, use the redis-di describe command (also available as redis-di status) to display its status. This combines the pipeline configuration with its runtime status, showing its overall state, its sources and targets, its jobs and components, and its per-stream statistics and performance metrics.

redis-di describe

To watch the status update live, pair the command with watch:

watch -n 1 redis-di describe

Pass a pipeline name, as in redis-di describe orders, to display a pipeline other than default. For a shorter overview, redis-di list prints a one-line summary of every pipeline, and redis-di get does the same for a single pipeline. See the redis-di describe reference page for details.

Start and stop a pipeline

Use redis-di stop to pause a running pipeline and redis-di start to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its configuration. A Debezium source restarts according to its snapshot mode and the availability of its saved source position.

redis-di stop
redis-di start

These commands, like reset and delete, act on the default pipeline unless you pass the name of another pipeline, as in redis-di stop orders.

To act on a single source instead of the whole pipeline, add --source:

redis-di stop --source mysql
redis-di start --source mysql

Note that a source can only run if its parent pipeline is running. See Multiple sources in one pipeline for more information.

Reset a pipeline

Use redis-di reset to clear the pipeline's saved source positions. With the default initial snapshot mode, a Debezium collector reloads selected source data and then resumes change data capture (CDC). Reset does not change the configured snapshot mode. Modes that skip existing rows do not reload them after a reset, and initial_only mode does not capture subsequent changes.

Reset does not flush the target Redis database. A new snapshot can overwrite target records, but missed deletes can leave stale records.

redis-di reset

Add --source to reset a single source. The other sources keep their data, but the whole pipeline stops while the reset runs and starts again afterwards:

redis-di reset --source mysql

If a reset fails, the pipeline stays in the error state until a later reset succeeds.

Clear a pipeline

To stop a pipeline and discard its configuration, deploy an empty configuration with the --empty option of redis-di deploy:

redis-di deploy --empty

RDI removes the pipeline's data plane components and deletes its data from the RDI database, including the change data streams, offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters of every source. The pipeline itself remains, with an empty configuration, so you can deploy a new configuration to it at any time.

The pipeline secrets are not affected, so remove them yourself with redis-di delete-secret if you no longer need them. The records the pipeline wrote to the target database are not deleted either.

Delete a pipeline

Use redis-di delete to delete a pipeline and its data plane components and keys from the RDI database. The command keeps the pipeline secrets and does not delete records already written to the target database. Pass --wait=false to return without waiting for the pipeline to disappear, or use --timeout to change the maximum wait time.

redis-di delete orders

Deleting a pipeline does not affect the other pipelines. See Multiple pipelines in one RDI installation for more information.

Wait for changes to complete

The commands that change a pipeline's state, namely deploy, start, stop, reset, delete, set-secret, and delete-secret, do not return as soon as the API accepts the request. By default, they wait for the pipeline to finish transitioning to the expected state, polling its status until it succeeds, reaches an error, or the --timeout (2 minutes by default) elapses. This is usually what you want: the command reflects the real outcome, so a script can rely on the change having taken effect and can fail fast if it did not.

In some cases, though, a pipeline needs several changes before it can transition to a healthy state, and waiting after each individual change would time out. The clearest example is rotating both the username and the password of a database: if you set only the username with the default --wait=true, the pipeline tries to reconnect with the new username and the old password, fails, and the command times out after two minutes with the pipeline in a broken state.

To avoid this, set all the related secrets, or at least all of them except the last, with --wait=false, so the pipeline applies them together and only the final command waits for it to become healthy:

redis-di set-secret USERNAME --db mysql newUsername --wait=false
redis-di set-secret PASSWORD --db mysql newPassword

The same applies to any set of changes that are only valid together.