{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/data-pipelines/multiple-pipelines",
  "title": "Multiple pipelines in one RDI installation",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-pipelines/",
  "summary": "Learn how to run several independent pipelines in one RDI installation.",
  "tags": [
    "docs",
    "integrate",
    "rs",
    "rdi"
  ],
  "last_updated": "2026-10-30T12:23:39+02:00",
  "page_type": "content",
  "content_hash": "02f724541ffb3a7b30a118f33b57103f3d5f4d67152f0dfd72fe426fe33ae5ce",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "One RDI installation can run several pipelines side by side. Each pipeline has its own\nconfiguration, jobs, secrets, and data plane components, which are its collectors and its\nstream processor. Each pipeline also keeps its own keys in the RDI database. You can deploy,\nstart, stop, reset, and delete a pipeline without affecting the others, and a pipeline that\nfails does not affect the others either. All the pipelines share the RDI control plane and the\nRDI database. See\n[How RDI is deployed](https://redis.io/docs/latest/integrate/redis-data-integration/architecture#how-rdi-is-deployed)\nfor a description of these components.\n\n> [!NOTE]\n> You must use the `redis-di` CLI or RDI API v2 to manage a pipeline other than `default`.\n> RDI API v1 manages only the `default` pipeline. See the\n> [RDI API migration guide](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-migration)\n> for more information."
    },
    {
      "id": "choose-between-pipelines-and-sources",
      "title": "Choose between pipelines and sources",
      "role": "content",
      "text": "A single pipeline can capture changes from several source databases. See\n[Multiple sources in one pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources).\nUse several sources in one pipeline when their data goes to the same target and you manage\nit as a unit. Use separate pipelines when:\n\n- The data goes to different target databases. A pipeline writes to a single target.\n- You want to stop, reset, or reconfigure some of the data without interrupting the rest.\n  Resetting or removing a single source stops the whole pipeline while the operation runs,\n  and RDI reports the whole pipeline in an error state when a single source fails.\n- You want separate processors for different sets of tables, so each processor handles its\n  own set of streams."
    },
    {
      "id": "name-your-pipelines",
      "title": "Name your pipelines",
      "role": "content",
      "text": "Every pipeline has a name, which identifies it in CLI commands, in API paths, and in the\nresources that RDI creates for it. A pipeline name must:\n\n- Be 3 to 20 characters long.\n- Start with a lowercase letter.\n- Contain only lowercase letters, digits, and dashes.\n- End with a letter or a digit.\n- Not contain two consecutive dashes.\n\nThe name `rdi` is reserved and cannot be used for a pipeline. You can't rename a pipeline.\nTo move a pipeline to a new name, export its configuration with\n[`redis-di export`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-export),\nset its secrets for the new pipeline, deploy the configuration as the new pipeline, and delete\nthe old one. The new pipeline starts with a new\n[initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle).\n\nThe pipeline named `default` is special in two ways:\n\n- The CLI targets it whenever you omit the pipeline name.\n- Its resources and keys have the same names as in earlier RDI versions, whether you created it\n  before or after upgrading to RDI 2.2.0, so an upgraded `default` pipeline keeps working\n  unchanged. Only a source that predates RDI's support for multiple sources keeps names of its\n  own. See\n  [Existing names are kept after an upgrade](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade).\n\nA new RDI installation has no pipeline, not even `default`, until you create one. See\n[Create a pipeline](#create-a-pipeline).\n\nThe optional [`metadata.name`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#metadata)\nproperty in `config.yaml` is a display name of up to 100 characters. It is not the pipeline\nname used in CLI commands or API paths.\n\nThe pipeline name appears in the resources that RDI creates for the pipeline. RDI adds the\npipeline name and two dashes to the name of each Kubernetes resource of a pipeline other than\n`default`, so pipeline, source, and secret provider names cannot contain consecutive dashes. The\nfollowing table lists the names for a pipeline named `orders` with a source named `mysql` and a\nsecret provider named `my-vault`, next to the names for the `default` pipeline. The `default`\nnames apply whether you created the `default` pipeline before or after the upgrade.\n\n| Resource | `default` pipeline | `orders` pipeline |\n| :-- | :-- | :-- |\n| Source credentials secret | `mysql-db` | `orders--mysql-db` |\n| Source TLS secret | `mysql-db-ssl` | `orders--mysql-db-ssl` |\n| Target credentials secret | `target-db` | `orders--target-db` |\n| Target TLS secret | `target-db-ssl` | `orders--target-db-ssl` |\n| SecretProviderClass | `my-vault` | `orders--my-vault` |\n| Collector deployment | `collector-mysql` | `orders--collector-mysql` |\n| Change data streams | `data:{rdi}:mysql.<qualified_table_name>` | `data:{<hash_tag>}:mysql.<qualified_table_name>` |\n| Dead-letter queue streams | `dlq:data:{rdi}:mysql.<qualified_table_name>` | `dlq:data:{<hash_tag>}:mysql.<qualified_table_name>` |\n| Collector metrics endpoint path on a VM installation | `/collector-mysql/metrics` | `/orders/collector-mysql/metrics` |\n| Flink processor metrics endpoint path on a VM installation | `/processor/metrics` | `/orders/processor/metrics` |\n| Classic processor metrics endpoint path on a VM installation | `/metrics` | `/orders/metrics-exporter/metrics` |\n| `pipeline` label of the collector and stream processor metrics on a Helm installation | `default` | `orders` |\n\nThe secret references in `config.yaml` do not contain the pipeline name. A source named\n`mysql` references `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}` in every pipeline, and\neach pipeline resolves them from its own secrets.\n\n`<hash_tag>` is the Redis [hash tag](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec#hash-tags)\nthat RDI assigns to the pipeline when you create it. The keys of a pipeline in the RDI database\nall share its hash tag. The hash tag of the `default` pipeline is always `rdi`, even when you\ncreate the `default` pipeline in RDI 2.2.0. For any other\npipeline, it is the pipeline name followed by an underscore and a short generated suffix, such\nas `orders_f2t`. RDI chooses the suffix so that new pipelines are spread approximately evenly\nacross the shards of a sharded RDI database. To find the hash tag of a pipeline, run:\n\n[code example]"
    },
    {
      "id": "create-a-pipeline",
      "title": "Create a pipeline",
      "role": "content",
      "text": "Create a pipeline with an empty configuration first, and then deploy its configuration with\n[`redis-di deploy`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-deploy):\n\n[code example]\n\nAn empty pipeline is stopped and has no collectors or stream processor. Creating it first lets\nRDI validate the configuration against the source and target databases when you deploy it.\nRunning `redis-di deploy orders --dir <path to pipeline folder>` before the pipeline exists\ncreates the pipeline, but doesn't run these database checks. If you omit the name, the commands\nact on the `default` pipeline. See\n[Deploy a pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#deploy-a-pipeline)\nfor the other `deploy` options.\n\nWith RDI API v2, create a pipeline with `POST /api/v2/pipelines`, passing the name in the\nrequest body, or with `PUT /api/v2/pipelines/{name}`. See the\n[API reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-reference) for details.\n\nDon't create pipelines with `kubectl`. Use the CLI or the API instead. RDI assigns the hash tag\nof a pipeline when the API creates it, so a `Pipeline` resource that you create directly with\n`kubectl` gets the hash tag of the `default` pipeline, and its keys collide with the keys of that\npipeline.\n\nIn a [high availability](https://redis.io/docs/latest/integrate/redis-data-integration/installation/install-vm#installing-with-high-availability)\nsetup, create pipelines on the primary instance. The secondary instance rejects requests that\ncreate a pipeline.\n\n> [!WARNING]\n> If you upgraded RDI from an earlier version, finish upgrading all the RDI instances before you\n> create a pipeline other than `default`, and do not downgrade RDI after you create one. See\n> [Upgrading to RDI 2.2.0](https://redis.io/docs/latest/integrate/redis-data-integration/installation/upgrade#upgrading-to-rdi-220)."
    },
    {
      "id": "set-secrets-for-each-pipeline",
      "title": "Set secrets for each pipeline",
      "role": "content",
      "text": "Every secret belongs to one pipeline. Name the pipeline with the `-p` or `--pipeline` option of\n[`redis-di set-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-set-secret):\n\n[code example]\n\nThe other secret commands, such as\n[`redis-di list-secrets`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-list-secrets)\nand [`redis-di delete-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete-secret),\ntake the same option. Without it, they manage the secrets of the `default` pipeline. See\n[Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)\nfor the available secret keys.\n\nYou can set the secrets of a pipeline before you create it, so that the pipeline can connect to\nits databases as soon as you deploy it. The secrets take effect once the pipeline exists. The\nonly exception is the scope-prefixed `SOURCE_DB_*` keys, which require an existing pipeline\nwith exactly one source. Deleting a pipeline does not delete its secrets. In a high availability\nsetup, set the secrets of every pipeline on both instances, because RDI does not replicate\nsecrets between them.\n\nIf you create the secrets of a pipeline other than `default` with `kubectl`, add the pipeline\nname and two dashes to the secret names, and set the `app.kubernetes.io/instance` label to the\npipeline name. The keys inside the secrets do not change. For example, for a source named\n`mysql` of the `orders` pipeline:\n\n[code example]\n\nSee [Set secrets for K8s/Helm deployment using Kubectl command](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command)\nfor the full list of secrets and keys.\n\nRDI creates a separate SecretProviderClass for each pipeline that declares a\n[secret provider](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#secret-providers),\nso two pipelines can declare secret providers with the same name. The `config.yaml` file\nstill refers to the provider by the name it has in the `secret-providers` section."
    },
    {
      "id": "select-the-pipeline-in-commands",
      "title": "Select the pipeline in commands",
      "role": "content",
      "text": "The pipeline commands\n[`deploy`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-deploy),\n[`start`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-start),\n[`stop`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-stop),\n[`reset`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-reset),\n[`delete`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete),\n[`get`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-get), and\n[`describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe)\ntake the pipeline name as an optional argument. The commands that manage the secrets,\ndead-letter queues, jobs, and metric collections of a pipeline take it with the `-p` or\n`--pipeline` option:\n\n[code example]\n\nAll these commands target the `default` pipeline when you omit the name, and there is no\nsetting to change this. Pass the name to every command that should act on another pipeline.\n[`redis-di list`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-list)\nlists all the pipelines with their status.\n\nRDI API v2 requests include the pipeline name in the path, as in\n`GET /api/v2/pipelines/{name}/status`. `GET /api/v2/pipelines` lists all the pipelines."
    },
    {
      "id": "start-stop-reset-and-delete-a-pipeline",
      "title": "Start, stop, reset, and delete a pipeline",
      "role": "content",
      "text": "Each lifecycle command acts only on the pipeline you name, while the other pipelines keep\nrunning:\n\n[code example]\n\nResetting a pipeline deletes only that pipeline's data from the RDI database and takes a new\n[initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle) of its\nsources. Add `--source` to act on a single source of the pipeline, for example\n`redis-di reset orders --source mysql`. See\n[Start, stop, and reset a single source](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources#start-stop-and-reset-a-single-source)\nfor details.\n\nUse [`redis-di delete`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete)\nto delete a pipeline. Any pipeline can be deleted, including `default`:\n\n[code example]\n\nRDI removes the pipeline's components and deletes its keys from the RDI database. The keys of\nthe other pipelines are not affected. The secrets of the pipeline are kept, so remove them with\n`redis-di delete-secret <key> --db <database> -p orders` if you no longer need them. The records\nthat the pipeline wrote to the target database are not deleted either. See\n[Delete a pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#delete-a-pipeline)\nfor the command options."
    },
    {
      "id": "write-to-the-same-or-different-targets",
      "title": "Write to the same or different targets",
      "role": "content",
      "text": "Each pipeline has a target database, which can be a separate database or the same database\nthat another pipeline writes to. When pipelines share a target database, make sure their jobs\nwrite to different keys, for example by using a different key prefix in each pipeline. RDI\nvalidates the jobs of each pipeline separately, so it does not detect keys that two pipelines\nboth write, and each pipeline overwrites the other's changes to such a key.\n\n> [!WARNING]\n> Flushing the target of a pipeline, for example with\n> `POST /api/v2/pipelines/{name}/flush-target/{target_name}`, deletes all the data in the target\n> database, including the records of every other pipeline that writes to it."
    },
    {
      "id": "monitor-each-pipeline",
      "title": "Monitor each pipeline",
      "role": "content",
      "text": "Use [`redis-di list`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-list)\nfor a one-line summary of every pipeline, and\n[`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe)\nwith a pipeline name for the details of one pipeline, including its components and their\nmetrics endpoints. Each pipeline has its own status, so a pipeline in an error state does not\nchange the status of the others.\n\nThe metrics of each pipeline are separate:\n\n- On a VM installation, the metrics endpoint paths of a pipeline other than `default` start\n  with the pipeline name, for example `/orders/collector-mysql/metrics`,\n  `/orders/processor/metrics` for the Flink processor, and `/orders/metrics-exporter/metrics`\n  for the classic processor. The metrics themselves carry no `pipeline` label. For example,\n  the `name` label of the Debezium collector metrics holds the source name, which can be the\n  same in several pipelines. When you create a pipeline, add a scrape job for each of its\n  endpoints to your Prometheus server, with a `pipeline` label, as shown in\n  [VM installation](https://redis.io/docs/latest/integrate/redis-data-integration/observability#vm-installation).\n- On a Helm installation, the ServiceMonitors of the Debezium collector, the Flink collector,\n  the Flink processor, and the classic processor metrics exporter add a `pipeline` label with\n  the pipeline name, for example `pipeline=\"orders\"`, or `pipeline=\"default\"` for the\n  `default` pipeline.\n- The `rdi_operator_pipeline_phase` operator metric reports the phase of each pipeline, with\n  the pipeline name in its `name` label.\n\nSee [Accessing the metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#accessing-the-metrics)\nfor the full list of endpoints.\n\nThe [support package](https://redis.io/docs/latest/integrate/redis-data-integration/observability#dump-support-package)\ncontains the diagnostics of every pipeline, in a separate `pipelines/<pipeline>` directory for\neach one."
    },
    {
      "id": "configure-ingresses-for-several-pipelines",
      "title": "Configure ingresses for several pipelines",
      "role": "configuration",
      "text": "On a Helm installation, the ingresses of the data plane components are disabled by default. The\ningress settings under `operator.dataPlane` in your Helm values file apply to every pipeline,\nsince there are no per-pipeline ingress settings. For a pipeline other than `default`, RDI adds\n`/<pipeline>` to the beginning of the path prefix of each data plane ingress, so the ingresses\nof different pipelines do not claim the same paths. For example, with the default path prefix\n`/processor`, the Flink processor ingress of the `orders` pipeline serves the Flink web UI at\n`/orders/processor`, and, if `operator.dataPlane.flinkProcessor.metricsAggregator.enabled` is\n`true`, its metrics at `/orders/processor/metrics`.\n\nAn empty or `/` path prefix stays unchanged, with one exception. The ingress of the classic\nprocessor metrics exporter, under `operator.dataPlane.metricsExporter`, has an empty path prefix\nby default. If you set no hosts for it, RDI replaces the empty prefix of a pipeline other than\n`default` with `/<pipeline>/metrics-exporter`, so the classic processor metrics of the `orders`\npipeline are at `/orders/metrics-exporter/metrics`.\n\nIf you use an empty or `/` path prefix together with hosts and more than one pipeline, include\nthe `$PIPELINE` variable in the ingress hosts to give each pipeline its own host:\n\n[code example]\n\nRDI replaces `$PIPELINE` with the pipeline name, so the host of the `orders` pipeline is\n`orders-processor.example.com`. For the `default` pipeline, RDI removes `$PIPELINE` together\nwith an adjacent dash, or else an adjacent dot, so its host is `processor.example.com`. The\n`$PIPELINE` variable is available in the ingress hosts of the Debezium collector, the Flink\ncollector, the Flink processor, and the classic processor metrics exporter, but not in their\npath prefixes.\n\nOn a VM installation, RDI configures these ingresses for you, so you do not need to change them."
    }
  ],
  "examples": [
    {
      "id": "name-your-pipelines-ex0",
      "language": "bash",
      "code": "kubectl -n rdi get pipeline orders -o jsonpath='{.spec.hashTag}'",
      "section_id": "name-your-pipelines"
    },
    {
      "id": "create-a-pipeline-ex0",
      "language": "bash",
      "code": "redis-di deploy orders --empty\nredis-di deploy orders --dir <path to pipeline folder>",
      "section_id": "create-a-pipeline"
    },
    {
      "id": "set-secrets-for-each-pipeline-ex0",
      "language": "bash",
      "code": "redis-di set-secret USERNAME --db mysql <username> -p orders\nredis-di set-secret PASSWORD --db mysql <password> -p orders\nredis-di set-secret PASSWORD --db target <password> -p orders",
      "section_id": "set-secrets-for-each-pipeline"
    },
    {
      "id": "set-secrets-for-each-pipeline-ex1",
      "language": "bash",
      "code": "kubectl create secret generic orders--mysql-db --namespace=rdi \\\n--from-literal=MYSQL_DB_USERNAME=yourUsername \\\n--from-literal=MYSQL_DB_PASSWORD=yourPassword \\\n--save-config --dry-run=client -o yaml | kubectl apply -f -\nkubectl label secret orders--mysql-db --namespace=rdi --overwrite \\\n  app.kubernetes.io/name=pipeline \\\n  app.kubernetes.io/instance=orders \\\n  product=rdi",
      "section_id": "set-secrets-for-each-pipeline"
    },
    {
      "id": "select-the-pipeline-in-commands-ex0",
      "language": "bash",
      "code": "redis-di describe orders\nredis-di list-dlqs -p orders\nredis-di list-jobs -p orders",
      "section_id": "select-the-pipeline-in-commands"
    },
    {
      "id": "start-stop-reset-and-delete-a-pipeline-ex0",
      "language": "bash",
      "code": "redis-di stop orders\nredis-di start orders\nredis-di reset orders",
      "section_id": "start-stop-reset-and-delete-a-pipeline"
    },
    {
      "id": "start-stop-reset-and-delete-a-pipeline-ex1",
      "language": "bash",
      "code": "redis-di delete orders",
      "section_id": "start-stop-reset-and-delete-a-pipeline"
    },
    {
      "id": "configure-ingresses-for-several-pipelines-ex0",
      "language": "yaml",
      "code": "operator:\n  dataPlane:\n    flinkProcessor:\n      ingress:\n        enabled: true\n        hosts:\n          - $PIPELINE-processor.example.com\n        pathPrefix: \"\"",
      "section_id": "configure-ingresses-for-several-pipelines"
    }
  ]
}