{ "schema_version": 2, "id": "integrate/redis-data-integration/data-pipelines/multiple-sources", "title": "Multiple sources in one pipeline", "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources/", "summary": "Learn how to ingest from several source databases into one Redis target.", "tags": [ "docs", "integrate", "rs", "rdi" ], "last_updated": "2026-09-18T16:39:51+01:00", "page_type": "content", "content_hash": "b58fb6749eafdf46ab99b3835e3ead666d30578f7865e8b82f5f968b869a46e6", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "One RDI pipeline can capture changes from several source databases and write them all to the\nsame Redis target. The sources can be of different database types and each has its\nown collector, its own credentials, and its own set of Redis streams to ensure it\nis independent of the other sources.\n\n> [!NOTE]\n> You must use RDI API v2 to manage a pipeline with several sources. RDI API v1 supports only\n> single-source pipelines. See the\n> [RDI API migration guide](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-migration) for more information." }, { "id": "name-your-sources", "title": "Name your sources", "role": "content", "text": "Each source is an entry in the `sources` section of\n[`config.yaml`](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config),\nwith the source name as key:\n\n[code example]\n\nA source name must:\n\n- Start with a lowercase letter.\n- Contain only lowercase letters, digits, and dashes.\n- End with a letter or a digit.\n- Be at most 22 characters long.\n\nThe names `rdi` and `target` are reserved and cannot be used for sources.\n\n> [!WARNING]\n> If your pipeline has a source created before RDI supported multiple sources, do not name a new\n> source after any schema or database of that older source. The change data streams of the older\n> source do not contain a source name segment, so a new source named after one of its schemas would claim keys\n> that belong to the older source, and resetting or removing the new source would delete the older\n> source's data. See\n> [Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade).\n\nRDI derives the environment variables that contain the source's credentials from the source\nname. For example, the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}`\nand `${MYSQL_DB_PASSWORD}`. See\n[Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)\nfor details of how RDI derives those names and for the full list of secret keys.\n\nThe source name also appears in the resources RDI creates for the source. The table below\nlists the names derived from a source named `mysql`.\n\n| Resource | Name |\n| :-- | :-- |\n| Credentials secret | `mysql-db` |\n| TLS secret | `mysql-db-ssl` |\n| Environment variable prefix | `MYSQL_DB_` |\n| Certificate directory | `/etc/certificates/mysql_db/` |\n| Collector deployment | `collector-mysql` |\n| Change data streams | `data:{rdi}:mysql.` |\n| Dead-letter queue streams | `dlq:data:{rdi}:mysql.` |\n| Metric collection | `collector-mysql_metrics` |\n| Metrics endpoint path on a VM installation | `/collector-mysql/metrics` |\n\nIn the stream names, `` is the qualified table name:\n`.` for MySQL and MariaDB, `.` for MongoDB,\n`.
` for Oracle, PostgreSQL, Snowflake, and Spanner, and\n`..
` for SQL Server.\n\nEach source also accepts an optional `name` property, which is a display name\nof up to 100 characters. Unlike the source name, it is not used as an identifier,\nso there is no restriction on the characters you can use." }, { "id": "configure-several-sources", "title": "Configure several sources", "role": "configuration", "text": "Add one entry per source in the `config.yaml` file (see\n[Pipeline configuration file](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config)\nfor a full description of this file).\n[`redis-di scaffold`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-scaffold)\ngenerates a configuration with one source, named by its `--source-name` option, so add any\nfurther sources by editing `config.yaml`.\n\nSources of different types can be mixed freely, but a source's collector `type` and its\n`connection` type have to match. Use `cdc`, the default, for the relational databases and\nMongoDB, `flink` for a Spanner connection, and `riotx` for a Snowflake connection. RDI\nrejects any other combination when you deploy the pipeline. See\n[Prepare source databases](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs)\nto learn how to prepare each source database.\n\nThe following example captures from a MySQL database and a PostgreSQL database, each with its\nown credentials:\n\n[code example]" }, { "id": "set-secrets-for-each-source", "title": "Set secrets for each source", "role": "content", "text": "Set a source's credentials with the source name in the `--db` option:\n\n[code example]\n\nThe secret keys used as CLI arguments are the same for every source: `USERNAME`, `PASSWORD`, and, for\n[Transport Layer Security (TLS)](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)\nconnections, `CACERT`, `CERT`, `KEY`, and `KEY_PASSWORD`. Use `--db target` for the\ntarget database: `redis-di set-secret PASSWORD --db target `.\n\nEach source's `connection` section then references its own secrets: `${MYSQL_DB_USERNAME}` and\n`${MYSQL_DB_PASSWORD}` for `mysql`, `${POSTGRESQL_DB_USERNAME}` and `${POSTGRESQL_DB_PASSWORD}`\nfor `postgresql`, and `${TARGET_DB_PASSWORD}` for the target.\n\nSee [Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)\nfor the full secret reference." }, { "id": "select-sources-in-jobs", "title": "Select sources in jobs", "role": "content", "text": "A [job](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples)\nselects the source it processes by setting `server_name` to the source name:\n\n[code example]\n\nWhen a pipeline has more than one source, every job must set `server_name`, and the value\nmust match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no\n`server_name`, or when its `server_name` matches no source. The one exception is the default\njob for `table: \"*\"`: it is a source-agnostic catch-all, so it needs no `server_name`.\n\nFor a source that existed before RDI supported multiple sources, set `server_name` to `rdi`\nrather than to the name the source has in `config.yaml`. See\n[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade).\n\nIn a pipeline with a single source, `server_name` is optional. If you omit it, the\njob does not filter by source.\n\nNo two jobs may select the same records, so make sure the source selectors of your jobs do\nnot overlap. RDI rejects the pipeline when it finds two jobs that intersect.\n\nWith the [Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/architecture/classic-vs-flink),\n`server_name` also accepts a list of source names, and an entry prefixed with `regex:` selects\nall sources that match the regular expression, so one job can process multiple tables,\npotentially from different sources, databases, or schemas. See\n[Job files](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples)\nfor details." }, { "id": "add-or-remove-a-source", "title": "Add or remove a source", "role": "content", "text": "To add a source, set its secrets first, then add it to `config.yaml` and deploy. Adding a\nsource does not interrupt other sources that are already running.\n\nTo remove a source, delete its entry from `config.yaml` and deploy. RDI removes the source's\ncollector and deletes that source's data from the RDI database, including its change data streams,\nDebezium offsets, schema history, dead-letter queue entries, statistics, deduplication state,\nand record counters. The other sources keep their data, and RDI stops the whole pipeline\nwhile the deletion runs and starts it again afterwards. No further action is\nneeded for this cleanup, but it means that a source you add later under the same name starts\nfrom a new\n[initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/architecture)\nrather than from the position it had reached.\n\nThe source's secrets are not deleted, so remove them yourself with\n[`redis-di delete-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete-secret)\nif you no longer need them. The records the pipeline wrote to the target database are not deleted\neither.\n\nAdding the same source again is straightforward, unless the source you removed predates RDI's support\nfor multiple sources. For such a source the names from before the upgrade are kept only while it\nexists, so any source you add under the same name is treated as a new source, for which RDI derives the\nnames instead. Adapt the configuration accordingly:\n\n- Set the source's secrets again, for example\n `redis-di set-secret PASSWORD --db mysql `.\n- Change the secret references in its `connection` section from `${SOURCE_DB_*}` to\n `${MYSQL_DB_*}`, for a source named `mysql`.\n- Change `server_name` from `rdi` to the source name in every job associated with it.\n\nSee\n[Redeploying a configuration after clearing a pipeline](#redeploying-a-configuration-after-clearing-a-pipeline)\nfor a before and after example, and\n[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade)\nfor the full list of names involved.\n\nNote that renaming a source is not supported. Renaming a source in `config.yaml` is equivalent\nto removing the source and adding a new source with the new name. This implies in particular:\n\n- You must create the source's secrets under the new name and update `${...}` references in\n its `connection` section.\n- You must update `server_name` for every job that reads from the source.\n- The data present in the RDI database under the old name is deleted, as it is for any removed source.\n- The source starts with a new\n [initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/architecture)." }, { "id": "start-stop-and-reset-a-single-source", "title": "Start, stop, and reset a single source", "role": "content", "text": "Pass `--source` to act on a single source instead of the whole pipeline:\n\n[code example]\n\nA source runs only while its pipeline runs, so starting one source does not start a stopped\npipeline. Generally, stopping one source leaves the others running, and when one source fails, the other sources keep capturing changes. The only exception to this is a source of type\n`external`. RDI creates no collector for this, so you cannot start or stop it.\n\nStopping a source scales its collector down to zero replicas and leaves the rest of the\nsource's resources in place. RDI records a captured position for each source, so when you restart a collector, it resumes from where it stopped.\n\nResetting a single source deletes that source's data from the RDI database, including its change data streams, Debezium\noffsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters.\nA new [initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/architecture) is then\ntaken for that source, while every other source keeps its data. RDI stops the whole pipeline while\nthe reset runs and starts it again afterwards, exactly as it does for a reset of the whole\npipeline." }, { "id": "monitor-each-source", "title": "Monitor each source", "role": "content", "text": "Use [`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe)\nto see the state of every source at once.\n\nIn its output, the `Sources` section lists each source with its sync mode and\nwhether it is connected.\nThe `Components` section lists one collector per source. Errors are reported against the\ncomponent they came from. See the\n[`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe)\nreference page for more details.\n\nNote that while the sources are independent of each other in the data they capture, the\npipeline status is not broken down per source. RDI reports the whole pipeline in an error state when a\nsingle source fails, so you should use the `Components` section to find out which one has failed.\n\nEach Debezium collector has its own metric collection, named after the collector, such as\n`collector-mysql_metrics`. The Flink and RIOT-X collectors don't have metric collections.\n\nIn Prometheus, you can break the per-stream record counters down per source, since the stream\nname contains the source name. With the\n[Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/architecture/classic-vs-flink)\nthe counters are reported by\n`flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`, which has a `stream`\nlabel; with the classic processor they are reported by `rdi_incoming_entries`, which has an\nequivalent `data_source` label. See\n[Flink processor metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#flink-processor-metrics),\n[Stream processor metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#stream-processor-metrics),\nand, for the per-source collector endpoints,\n[Accessing the metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#accessing-the-metrics).\n\nDead-letter queue streams have Redis keys containing a\n`.` section.\nThis makes it easy to attribute rejected records to their source. See\n[Rejected records](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/rejected-records) for more information." }, { "id": "existing-names-are-kept-after-an-upgrade", "title": "Existing names are kept after an upgrade", "role": "content", "text": "Before RDI supported multiple sources per pipeline, every source-scoped resource had\na name including the word `source` instead of the actual source name in `config.yaml`.\nFor a source that existed before you upgraded to a version that supports multiple sources,\nthose names are kept unchanged, regardless of what the source is called in `config.yaml`.\nIn particular, for such a source:\n\n- Its secret environment variables are still named `SOURCE_DB_*`, so its `connection` section\n can keep referencing these secrets.\n- Its Kubernetes secrets are still named `source-db` and `source-db-ssl`.\n- Its Kubernetes deployment and other resources are still named `collector-source`.\n- Its data streams are still named `data:{rdi}:`, and its offset\n and schema history keys are still `metadata:debezium:offsets` and\n `metadata:debezium:schema_history`.\n- Its `server_name` is still `rdi`, or, for a Spanner source, its instance ID.\n\nRDI keeps these names in a mapping from the source name in `config.yaml` to\nthe internal name the source had before the upgrade. This mapping lasts only as long as the source\ndoes: RDI discards it as soon as the source is removed from the configuration, whether you remove that one\nsource or\n[clear the whole pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline).\nA source you add afterwards under the same name is treated as a new source, so\nRDI derives its names from the source name. See\n[Add or remove a source](#add-or-remove-a-source) for what you have to change in that case.\n\nFor a source you add after the upgrade under any other name, RDI derives all of these names\nfrom the source name, as described on this page.\n\nSee [Upgrading RDI](https://redis.io/docs/latest/integrate/redis-data-integration/installation/upgrade)\nfor more information." }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline", "title": "Redeploying a configuration after clearing a pipeline", "role": "content", "text": "A configuration exported from an upgraded pipeline still references the names from before the\nupgrade, so deploying it again after\n[clearing the pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline)\nfails, because the mapping that made those names resolve is gone.\n\nA source and a job of such an upgraded pipeline:\n\n[code example]\n\n[code example]\n\nThe same source and job, adapted to deploy as a new source named `mysql`:\n\n[code example]\n\n[code example]\n\nSet the source's secrets under its actual name before you deploy:\n\n[code example]\n\nThe source then takes a fresh\n[initial snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/architecture), because the position it had\nreached was deleted along with the rest of its data. Records the pipeline already wrote to the\ntarget database are not deleted, so the snapshot overwrites them." } ], "examples": [ { "id": "name-your-sources-ex0", "language": "yaml", "code": "sources:\n mysql: # this source is named 'mysql'\n type: cdc", "section_id": "name-your-sources" }, { "id": "configure-several-sources-ex0", "language": "yaml", "code": "sources:\n mysql:\n type: cdc\n connection:\n type: mysql\n host: \n port: 3306\n user: ${MYSQL_DB_USERNAME}\n password: ${MYSQL_DB_PASSWORD}\n databases:\n - inventory\n tables:\n inventory.customers: {}\n inventory.orders: {}\n postgresql:\n type: cdc\n connection:\n type: postgresql\n host: \n port: 5432\n database: billing\n user: ${POSTGRESQL_DB_USERNAME}\n password: ${POSTGRESQL_DB_PASSWORD}\n schemas:\n - public\n tables:\n public.clients: {}\ntargets:\n target:\n connection:\n type: redis\n host: \n port: 6379\n password: ${TARGET_DB_PASSWORD}", "section_id": "configure-several-sources" }, { "id": "set-secrets-for-each-source-ex0", "language": "bash", "code": "redis-di set-secret USERNAME --db mysql \nredis-di set-secret PASSWORD --db mysql \nredis-di set-secret USERNAME --db postgresql \nredis-di set-secret PASSWORD --db postgresql ", "section_id": "set-secrets-for-each-source" }, { "id": "select-sources-in-jobs-ex0", "language": "yaml", "code": "source:\n server_name: mysql\n db: inventory\n table: customers", "section_id": "select-sources-in-jobs" }, { "id": "start-stop-and-reset-a-single-source-ex0", "language": "bash", "code": "redis-di stop --source mysql\nredis-di start --source mysql\nredis-di reset --source mysql", "section_id": "start-stop-and-reset-a-single-source" }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline-ex0", "language": "yaml", "code": "sources:\n mysql:\n connection:\n user: ${SOURCE_DB_USERNAME}\n password: ${SOURCE_DB_PASSWORD}", "section_id": "redeploying-a-configuration-after-clearing-a-pipeline" }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline-ex1", "language": "yaml", "code": "source:\n server_name: rdi\n db: inventory\n table: customers", "section_id": "redeploying-a-configuration-after-clearing-a-pipeline" }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline-ex2", "language": "yaml", "code": "sources:\n mysql:\n connection:\n user: ${MYSQL_DB_USERNAME}\n password: ${MYSQL_DB_PASSWORD}", "section_id": "redeploying-a-configuration-after-clearing-a-pipeline" }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline-ex3", "language": "yaml", "code": "source:\n server_name: mysql\n db: inventory\n table: customers", "section_id": "redeploying-a-configuration-after-clearing-a-pipeline" }, { "id": "redeploying-a-configuration-after-clearing-a-pipeline-ex4", "language": "bash", "code": "redis-di set-secret USERNAME --db mysql \nredis-di set-secret PASSWORD --db mysql ", "section_id": "redeploying-a-configuration-after-clearing-a-pipeline" } ] }