{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/data-pipelines",
  "title": "Data pipelines",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/",
  "summary": "Learn how to configure RDI for data capture and transformation.",
  "content": "\nRDI uses *pipelines* to implement\n[change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC). (See the\n[architecture overview](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture#overview)\nfor an introduction to pipelines.)\nThe sections below explain how pipelines work and give an overview of how to configure and\ndeploy them.\n\n## How a pipeline works\n\nAn RDI pipeline captures change data records from the source database, and transforms them\ninto Redis data structures. It writes each of these new structures to a Redis target\ndatabase under its own key. \n\nBy default, RDI transforms the source data into\n[hashes](https://redis.io/docs/latest/develop/data-types/hashes) or\n[JSON objects](https://redis.io/docs/latest/develop/data-types/json) for the target with a\nstandard data mapping and a standard format for the key.\nHowever, you can also provide your own custom transformation [jobs](#job-files)\nfor each source table, using your own data mapping and key pattern. You specify these\njobs declaratively with YAML configuration files that require no coding.\n\nData transformation involves two stages:\n\n1.  The data ingested during CDC is automatically transformed to an intermediate JSON\n    change event format.\n1.  RDI passes this JSON change event data to your custom transformation for further\n    processing.\n\nThe diagram below shows the flow of data through the pipeline:\n\n![images/rdi/ingest/RDIPipeDataflow.webp](https://redis.io/docs/latest/images/rdi/ingest/RDIPipeDataflow.webp)\n\nYou can provide a job file for each source table that needs a custom\ntransformation. You can also add a *default job file* for any tables that don't have their own.\nYou must specify the full name of the source table in the job file (or the special\nname \"*\" in the default job) and you\ncan also include filtering logic to skip data that matches a particular condition.\nAs part of the transformation, you can specify any of the following data types\nto store the data in Redis:\n\n- [JSON](https://redis.io/docs/latest/develop/data-types/json)\n- [Hashes](https://redis.io/docs/latest/develop/data-types/hashes)\n- [Sets](https://redis.io/docs/latest/develop/data-types/sets)\n- [Streams](https://redis.io/docs/latest/develop/data-types/streams)\n- [Sorted sets](https://redis.io/docs/latest/develop/data-types/sorted-sets)\n- [Strings](https://redis.io/docs/latest/develop/data-types/strings)\n\n### Pipeline lifecycle\n\nAfter you deploy a pipeline, it goes through the following phases:\n\n1. *Deploy* - when you deploy the pipeline, RDI first validates it before use.\nThen, the [operator](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture#how-rdi-is-deployed) creates and configures the collector and stream processor that will run the pipeline.\n1. *Snapshot* - The collector starts the pipeline by creating a snapshot of the full\ndataset. This involves reading all the relevant source data, transforming it and then\nwriting it into the Redis target. This phase typically takes minutes to\nhours if you have a lot of data.\n1. *CDC* - Once the snapshot is complete, the collector starts listening for updates to\nthe source data. Whenever a change is committed to the source, the collector captures\nit and adds it to the target through the pipeline. This phase continues indefinitely\nunless you change the pipeline configuration. \n1. *Update* - If you update the pipeline configuration, the operator applies it\nto the collector and the stream processor. Note that the changes only affect newly-captured\ndata unless you reset the pipeline completely. Once RDI has accepted the updates, the\npipeline returns to the CDC phase with the new configuration.\n1. *Reset* - There are circumstances where you might want to rebuild the dataset\ncompletely. For example, you might want to apply a new transformation to all the source\ndata or refresh the dataset if RDI is disconnected from the\nsource for a long time. In situations like these, you can *reset* the pipeline back\nto the snapshot phase. When this is complete, the pipeline continues with CDC as usual. \n\n## Using a pipeline\n\nFollow the steps described in the sections below to prepare and run an RDI pipeline.\n\n### 1. Prepare the source database\n\nBefore using the pipeline you must first prepare your source database to use\nthe Debezium connector for *change data capture (CDC)*. See the\n[architecture overview](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture#overview)\nfor more information about CDC.\nEach database type has a different set of preparation steps. You can\nfind the preparation guides for the databases that RDI supports in the\n[Prepare source databases](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs)\nsection.\n\n###  2. Configure the pipeline\n\nRDI uses a set of [YAML](https://en.wikipedia.org/wiki/YAML)\nfiles to configure each pipeline. The folder structure of the\nconfiguration is shown below:\n\n```hierarchy {type=\"filesystem\"}\n\"(root)\":\n    \"config.yaml\":\n        _meta:\n            description: \"\\\"config.yaml\\\" is the main pipeline configuration file.\"\n    \"jobs\":\n        _meta:\n            description: \"The 'jobs' folder containing optional job files.\"\n        \"default-job.yaml\":\n            _meta:\n                description: \"A default job.\"\n        \"job1.yaml\":\n          _meta:\n                description: \"Each job file must have a unique name.\"\n        \"...\":\n            _meta:\n                ellipsis: true\n                description: \"Other job files, if required.\"\n```\n\nThe main configuration for the pipeline is in the `config.yaml` file.\nThis specifies the connection details for the source database (such\nas host, username, and password) and also the queries that RDI will use\nto extract the required data. You should place job files in the `Jobs`\nfolder if you want to specify your own data transformations.\n\nSee\n[Pipeline configuration file](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config)\nfor a full description of the `config.yaml` file and some example configurations.\n\n### 3. Create job files (optional)\n\nYou can use one or more job files to configure which fields from the source tables\nyou want to use, and which data structure you want to write to the target. You\ncan also optionally specify a transformation to apply to the data before writing it\nto the target. See the\n[Job files](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples)\nsection for full details of the file format and examples of common tasks for job files.\n\n### 4. Deploy the pipeline\n\nWhen your configuration is ready, you must deploy it to start using the pipeline. See\n[Deploy a pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy)\nto learn how to do this.\n\n## More information\n\nSee the other pages in this section for more information and examples:\n",
  "tags": ["docs","integrate","rs","rdi"],
  "last_updated": "2026-09-19T17:55:58-07:00",
  "children": [{"id":"integrate/redis-data-integration/data-pipelines/prepare-dbs","summary":"Enable CDC features in your source databases","title":"Prepare source databases","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/"},{"id":"integrate/redis-data-integration/data-pipelines/pipeline-config","summary":"Learn how to specify the main configuration details for an RDI pipeline.","title":"Pipeline configuration file","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config/"},{"id":"integrate/redis-data-integration/data-pipelines/transform-examples","summary":"Learn how to configure job files for data transformation.","title":"Job files","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/"},{"id":"integrate/redis-data-integration/data-pipelines/data-denormalization","summary":"Learn about denormalization strategies","title":"Data denormalization","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization/"},{"id":"integrate/redis-data-integration/data-pipelines/rejected-records","summary":"Learn how RDI stores records that cannot be processed.","title":"Rejected records","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records/"},{"id":"integrate/redis-data-integration/data-pipelines/deploy","summary":"Learn how to deploy an RDI pipeline","title":"Deploy a pipeline","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy/"},{"id":"integrate/redis-data-integration/data-pipelines/supported-types","summary":"Learn about supported data types for each source database.","title":"Supported data types by source","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/supported-types/"}]
}