Multiple pipelines in one RDI installation

Learn how to run several independent pipelines in one RDI installation.

One RDI installation can run several pipelines side by side. Each pipeline has its own configuration, jobs, secrets, and data plane components, which are its collectors and its stream processor. Each pipeline also keeps its own keys in the RDI database. You can deploy, start, stop, reset, and delete a pipeline without affecting the others, and a pipeline that fails does not affect the others either. All the pipelines share the RDI control plane and the RDI database. See How RDI is deployed for a description of these components.

Choose between pipelines and sources

A single pipeline can capture changes from several source databases. See Multiple sources in one pipeline. Use several sources in one pipeline when their data goes to the same target and you manage it as a unit. Use separate pipelines when:

• The data goes to different target databases. A pipeline writes to a single target.
• You want to stop, reset, or reconfigure some of the data without interrupting the rest. Resetting or removing a single source stops the whole pipeline while the operation runs, and RDI reports the whole pipeline in an error state when a single source fails.
• You want separate processors for different sets of tables, so each processor handles its own set of streams.

Name your pipelines

Every pipeline has a name, which identifies it in CLI commands, in API paths, and in the resources that RDI creates for it. A pipeline name must:

• Be 3 to 20 characters long.
• Start with a lowercase letter.
• Contain only lowercase letters, digits, and dashes.
• End with a letter or a digit.
• Not contain two consecutive dashes.

The name rdi is reserved and cannot be used for a pipeline. You can't rename a pipeline. To move a pipeline to a new name, export its configuration with redis-di export, set its secrets for the new pipeline, deploy the configuration as the new pipeline, and delete the old one. The new pipeline starts with a new initial snapshot.

The pipeline named default is special in two ways:

• The CLI targets it whenever you omit the pipeline name.
• Its resources and keys have the same names as in earlier RDI versions, whether you created it before or after upgrading to RDI 2.2.0, so an upgraded default pipeline keeps working unchanged. Only a source that predates RDI's support for multiple sources keeps names of its own. See Existing names are kept after an upgrade.

A new RDI installation has no pipeline, not even default, until you create one. See Create a pipeline.

The optional metadata.name property in config.yaml is a display name of up to 100 characters. It is not the pipeline name used in CLI commands or API paths.

The pipeline name appears in the resources that RDI creates for the pipeline. RDI adds the pipeline name and two dashes to the name of each Kubernetes resource of a pipeline other than default, so pipeline, source, and secret provider names cannot contain consecutive dashes. The following table lists the names for a pipeline named orders with a source named mysql and a secret provider named my-vault, next to the names for the default pipeline. The default names apply whether you created the default pipeline before or after the upgrade.

The secret references in config.yaml do not contain the pipeline name. A source named mysql references ${MYSQL_DB_USERNAME} and ${MYSQL_DB_PASSWORD} in every pipeline, and each pipeline resolves them from its own secrets.

<hash_tag> is the Redis hash tag that RDI assigns to the pipeline when you create it. The keys of a pipeline in the RDI database all share its hash tag. The hash tag of the default pipeline is always rdi, even when you create the default pipeline in RDI 2.2.0. For any other pipeline, it is the pipeline name followed by an underscore and a short generated suffix, such as orders_f2t. RDI chooses the suffix so that new pipelines are spread approximately evenly across the shards of a sharded RDI database. To find the hash tag of a pipeline, run:

kubectl -n rdi get pipeline orders -o jsonpath='{.spec.hashTag}'

Create a pipeline

Create a pipeline with an empty configuration first, and then deploy its configuration with redis-di deploy:

redis-di deploy orders --empty
redis-di deploy orders --dir <path to pipeline folder>

An empty pipeline is stopped and has no collectors or stream processor. Creating it first lets RDI validate the configuration against the source and target databases when you deploy it. Running redis-di deploy orders --dir <path to pipeline folder> before the pipeline exists creates the pipeline, but doesn't run these database checks. If you omit the name, the commands act on the default pipeline. See Deploy a pipeline for the other deploy options.

With RDI API v2, create a pipeline with POST /api/v2/pipelines, passing the name in the request body, or with PUT /api/v2/pipelines/{name}. See the API reference for details.

Don't create pipelines with kubectl. Use the CLI or the API instead. RDI assigns the hash tag of a pipeline when the API creates it, so a Pipeline resource that you create directly with kubectl gets the hash tag of the default pipeline, and its keys collide with the keys of that pipeline.

In a high availability setup, create pipelines on the primary instance. The secondary instance rejects requests that create a pipeline.

Set secrets for each pipeline

Every secret belongs to one pipeline. Name the pipeline with the -p or --pipeline option of redis-di set-secret:

redis-di set-secret USERNAME --db mysql <username> -p orders
redis-di set-secret PASSWORD --db mysql <password> -p orders
redis-di set-secret PASSWORD --db target <password> -p orders

The other secret commands, such as redis-di list-secrets and redis-di delete-secret, take the same option. Without it, they manage the secrets of the default pipeline. See Set secrets for the available secret keys.

You can set the secrets of a pipeline before you create it, so that the pipeline can connect to its databases as soon as you deploy it. The secrets take effect once the pipeline exists. The only exception is the scope-prefixed SOURCE_DB_* keys, which require an existing pipeline with exactly one source. Deleting a pipeline does not delete its secrets. In a high availability setup, set the secrets of every pipeline on both instances, because RDI does not replicate secrets between them.

If you create the secrets of a pipeline other than default with kubectl, add the pipeline name and two dashes to the secret names, and set the app.kubernetes.io/instance label to the pipeline name. The keys inside the secrets do not change. For example, for a source named mysql of the orders pipeline:

kubectl create secret generic orders--mysql-db --namespace=rdi \
--from-literal=MYSQL_DB_USERNAME=yourUsername \
--from-literal=MYSQL_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -
kubectl label secret orders--mysql-db --namespace=rdi --overwrite \
  app.kubernetes.io/name=pipeline \
  app.kubernetes.io/instance=orders \
  product=rdi

See Set secrets for K8s/Helm deployment using Kubectl command for the full list of secrets and keys.

RDI creates a separate SecretProviderClass for each pipeline that declares a secret provider, so two pipelines can declare secret providers with the same name. The config.yaml file still refers to the provider by the name it has in the secret-providers section.

Select the pipeline in commands

The pipeline commands deploy, start, stop, reset, delete, get, and describe take the pipeline name as an optional argument. The commands that manage the secrets, dead-letter queues, jobs, and metric collections of a pipeline take it with the -p or --pipeline option:

redis-di describe orders
redis-di list-dlqs -p orders
redis-di list-jobs -p orders

All these commands target the default pipeline when you omit the name, and there is no setting to change this. Pass the name to every command that should act on another pipeline. redis-di list lists all the pipelines with their status.

RDI API v2 requests include the pipeline name in the path, as in GET /api/v2/pipelines/{name}/status. GET /api/v2/pipelines lists all the pipelines.

Start, stop, reset, and delete a pipeline

Each lifecycle command acts only on the pipeline you name, while the other pipelines keep running:

redis-di stop orders
redis-di start orders
redis-di reset orders

Resetting a pipeline deletes only that pipeline's data from the RDI database and takes a new initial snapshot of its sources. Add --source to act on a single source of the pipeline, for example redis-di reset orders --source mysql. See Start, stop, and reset a single source for details.

Use redis-di delete to delete a pipeline. Any pipeline can be deleted, including default:

redis-di delete orders

RDI removes the pipeline's components and deletes its keys from the RDI database. The keys of the other pipelines are not affected. The secrets of the pipeline are kept, so remove them with redis-di delete-secret <key> --db <database> -p orders if you no longer need them. The records that the pipeline wrote to the target database are not deleted either. See Delete a pipeline for the command options.

Write to the same or different targets

Each pipeline has a target database, which can be a separate database or the same database that another pipeline writes to. When pipelines share a target database, make sure their jobs write to different keys, for example by using a different key prefix in each pipeline. RDI validates the jobs of each pipeline separately, so it does not detect keys that two pipelines both write, and each pipeline overwrites the other's changes to such a key.

Monitor each pipeline

Use redis-di list for a one-line summary of every pipeline, and redis-di describe with a pipeline name for the details of one pipeline, including its components and their metrics endpoints. Each pipeline has its own status, so a pipeline in an error state does not change the status of the others.

The metrics of each pipeline are separate:

• On a VM installation, the metrics endpoint paths of a pipeline other than default start with the pipeline name, for example /orders/collector-mysql/metrics, /orders/processor/metrics for the Flink processor, and /orders/metrics-exporter/metrics for the classic processor. The metrics themselves carry no pipeline label. For example, the name label of the Debezium collector metrics holds the source name, which can be the same in several pipelines. When you create a pipeline, add a scrape job for each of its endpoints to your Prometheus server, with a pipeline label, as shown in VM installation.
• On a Helm installation, the ServiceMonitors of the Debezium collector, the Flink collector, the Flink processor, and the classic processor metrics exporter add a pipeline label with the pipeline name, for example pipeline="orders", or pipeline="default" for the default pipeline.
• The rdi_operator_pipeline_phase operator metric reports the phase of each pipeline, with the pipeline name in its name label.

See Accessing the metrics for the full list of endpoints.

The support package contains the diagnostics of every pipeline, in a separate pipelines/<pipeline> directory for each one.

Configure ingresses for several pipelines

On a Helm installation, the ingresses of the data plane components are disabled by default. The ingress settings under operator.dataPlane in your Helm values file apply to every pipeline, since there are no per-pipeline ingress settings. For a pipeline other than default, RDI adds /<pipeline> to the beginning of the path prefix of each data plane ingress, so the ingresses of different pipelines do not claim the same paths. For example, with the default path prefix /processor, the Flink processor ingress of the orders pipeline serves the Flink web UI at /orders/processor, and, if operator.dataPlane.flinkProcessor.metricsAggregator.enabled is true, its metrics at /orders/processor/metrics.

An empty or / path prefix stays unchanged, with one exception. The ingress of the classic processor metrics exporter, under operator.dataPlane.metricsExporter, has an empty path prefix by default. If you set no hosts for it, RDI replaces the empty prefix of a pipeline other than default with /<pipeline>/metrics-exporter, so the classic processor metrics of the orders pipeline are at /orders/metrics-exporter/metrics.

If you use an empty or / path prefix together with hosts and more than one pipeline, include the $PIPELINE variable in the ingress hosts to give each pipeline its own host:

operator:
  dataPlane:
    flinkProcessor:
      ingress:
        enabled: true
        hosts:
          - $PIPELINE-processor.example.com
        pathPrefix: ""

RDI replaces $PIPELINE with the pipeline name, so the host of the orders pipeline is orders-processor.example.com. For the default pipeline, RDI removes $PIPELINE together with an adjacent dash, or else an adjacent dot, so its host is processor.example.com. The $PIPELINE variable is available in the ingress hosts of the Debezium collector, the Flink collector, the Flink processor, and the classic processor metrics exporter, but not in their path prefixes.

On a VM installation, RDI configures these ingresses for you, so you do not need to change them.