{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/data-pipelines/pipeline-config",
  "title": "Pipeline configuration file",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config/",
  "summary": "Learn how to specify the main configuration details for an RDI pipeline.",
  "content": "\nThe main configuration details for an RDI pipeline are in the `config.yaml` file.\nThis file specifies the connection details for the source and target databases,\nand also the set of tables you want to capture. You can also add one or more\n[job files](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples)\nif you want to apply custom transformations to the captured data.\n\nEach section explains one part of the file. Start with the minimal example, then\nadd only the optional properties that you need. See the\n[configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference)\nfor all supported properties.\n\n## Before you start\n\nBefore you create `config.yaml`:\n\n1. [Prepare the source database](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs) for change data capture.\n1. [Install RDI](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation).\n1. [Set the secrets](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets) that the file references.\n\n## Start with a minimal file\n\nThe following example shows the required structure of a `config.yaml` file. Values of the\nform \"`${name}`\" refer to secrets that you should set as described in\n[Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets).\nIn particular, you should normally use secrets as shown to set the source\nand target username and password rather than storing them in plain text in this file.\n\n```yaml\nsources:\n  mysql:\n    type: cdc\n    logging:\n      level: info\n    connection:\n      type: mysql\n      host: \u003cDB_HOST\u003e\n      port: 3306\n      user: ${SOURCE_DB_USERNAME}\n      password: ${SOURCE_DB_PASSWORD}\n\ntargets:\n  target:\n    connection:\n      type: redis\n      host: \u003cREDIS_TARGET_DB_HOST\u003e\n      port: \u003cREDIS_TARGET_DB_PORT\u003e\n      password: ${TARGET_DB_PASSWORD}\n\nprocessors:\n  type: flink\n  target_data_type: hash\n```\n\nKeep `type: flink` for new pipelines. The other processor properties have defaults,\nso add them only when you need to change the default behavior.\n\n## Build the file with an AI assistant\n\nCopy the following prompt into your AI assistant. The prompt tells the assistant\nto use the RDI documentation as its source of truth and to flag unsupported requests.\n\n```text\nHelp me create a valid Redis Data Integration (RDI) config.yaml file.\n\nUse only these pages as sources for configuration properties and behavior:\n- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/\n- https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/\n- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/\n\nDo not invent property names. If a requested property is not documented, tell me.\nUse ${NAME} secret references for credentials and certificates. Do not include secret\nvalues in the file. Configure one target Redis database named `target`. Always use the\nFlink processor by setting `processors.type` to `flink`.\n\nAsk me for the following information one question at a time:\n1. Source database type, host, and port.\n2. Databases or schemas to capture.\n3. Tables and columns to capture, including keys for tables without a primary key\n   or unique constraint.\n4. Whether the initial snapshot needs a row filter.\n5. Target Redis host and port, and whether the connection uses TLS or mTLS.\n6. Redis hash or JSON output.\n\nAfter I answer, generate config.yaml. Then list the required secrets and link me to\nthe documented commands to set the secrets and deploy the pipeline.\n```\n\n## Sections\n\nThe main sections of the file configure [`sources`](#sources), [`targets`](#targets),\nand [`processors`](#processors).\n\n### Sources\n\nThe `sources` section has a subsection for the source that\nyou need to configure. The source section starts with a unique name\nto identify the source (in the example, there is a source\ncalled `mysql` but you can choose any name you like). The example\nconfiguration contains the following data:\n\n- `type`: The collector to use for the pipeline. Use `cdc` for MariaDB, MySQL,\n  MongoDB, Oracle, PostgreSQL, or SQL Server. Use `flink` for Google Cloud\n  Spanner. Use `riotx` for Snowflake. Use `external` when you provide and manage\n  the collector. RDI doesn't create collector resources for an `external` source,\n  so omit the other properties in the source section.\n- `connection`: The connection details for the source database: `type`, `host`, `port`,\n  and credentials (`user` and `password`).\n  See the [configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#sourcesconnection)\n  for the required fields for each source database type.\n  - If you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/\n    or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect\n    to the source database, you may need to specify additional properties in the\n    `advanced` section with references to the corresponding certificates depending\n    on the source database type. Note that these properties **must** be references to\n    secrets that you should set as described in [Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets).\n- `databases`: List of all databases to collect data from for source database types\n  that support multiple databases, such as `mysql` and `mariadb`.\n- `schemas`: List of all schemas to collect data from for source database types\n  that support multiple schemas, such as `oracle`, `postgresql`, and `sqlserver`.\n- `tables`: List of all tables to collect data from. Each table is identified by its\n  full name, including a database or schema prefix. If there is a single\n  database or schema, this prefix can be omitted.\n  For each table, you can specify:\n  - `columns`: A list of the columns you are interested in (the default is to\n    include all columns)\n  - `keys`: A list of columns to create a composite key if your table\n    doesn't already have a [`PRIMARY KEY`](https://www.w3schools.com/sql/sql_primarykey.asp) or\n    [`UNIQUE`](https://www.w3schools.com/sql/sql_unique.asp) constraint.\n  - `snapshot_sql`: A query to be used when performing the initial snapshot.\n    By default, a query that contains all listed columns of all listed tables will be used.\n- `advanced`: These optional properties configure other Debezium-specific features.\n  The available sub-sections are:\n  - `source`: Properties for reading from the source database.\n    See the Debezium [Source connectors](https://debezium.io/documentation/reference/stable/connectors/)\n    pages for more information about the properties available for each database type.\n  - `sink`: Properties for writing to Redis streams in the RDI database.\n    See the Debezium [Redis stream properties](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream)\n    page for the full set of available properties.\n  - `quarkus`: Properties for the Debezium server, such as the log level. See the\n    Quarkus [Configuration options](https://quarkus.io/guides/all-config)\n    docs for the full set of available properties.\n  - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server.\n    For example, set it to `\"-Xmx2g -Xms512m\"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB.\n\n### Targets\n\nUse this section to provide the connection details for the target Redis\ndatabase. RDI supports one target database. Name the target `target`.\nIn the `connection` section, you can specify the\n`type` of the target database, which must be `redis`, along with\nconnection details such as `host`, `port`, and credentials (`user` and `password`).\nIf you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/\nor [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect\nto the target database, you must specify the CA certificate (for TLS),\nand the client certificate and private key (for mTLS) in `cacert`, `cert`, and `key`.\nNote that these certificates **must** be references to secrets\nthat you should set as described in [Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets)\n(it is not possible to include these certificates as plain text in the file).\n\n### Processors\n\nThe `processors` section selects the stream processor and configures its behavior.\nUse the Flink processor for new pipelines:\n\n```yaml\nprocessors:\n  type: flink\n```\n\nSee [Differences between the classic and Flink processors](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink)\nand [Migrate from the classic processor to the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink)\nfor existing pipelines.\n\n### Tune Classic processor performance\n\nThe Classic processor uses the top-level batch, queue, initial-sync, and stream\npolling properties. Larger batches can improve throughput but use more memory and\ncan increase latency while RDI waits for a batch to fill.\n\n```yaml\nprocessors:\n  type: classic\n  read_batch_size: 2000\n  read_batch_timeout_ms: 100\n  write_batch_size: 200\n  enable_async_processing: true\n  batch_queue_size: 3\n  ack_queue_size: 10\n  initial_sync_processes: 4\n  idle_sleep_time_ms: 200\n  idle_streams_check_interval_ms: 1000\n  busy_streams_check_interval_ms: 5000\n```\n\n### Tune Flink processor performance\n\nThe Flink processor uses `processors.advanced` for batch behavior, parallelism,\nand memory. Don't use Classic queue and initial-sync properties to tune Flink.\n\n```yaml\nprocessors:\n  type: flink\n  advanced:\n    source:\n      batch.size: 2000\n      batch.timeout.ms: 100\n      discovery.interval.ms: 1000\n    target:\n      batch.size: 200\n      flush.interval.ms: 100\n    flink:\n      taskmanager.numberOfTaskSlots: 1\n      taskmanager.memory.process.size: 2048m\n    resources:\n      taskManager:\n        replicas: 2\n```\n\nFor Kubernetes installations, the number of available task slots is the number\nof TaskManager replicas multiplied by `taskmanager.numberOfTaskSlots`. When you\nomit `parallelism.default`, Flink uses the available task slots. Adding task slots\ncan increase initial snapshot throughput. Size `taskmanager.memory.process.size`\nfor the work done by each TaskManager, especially when jobs use transformations.\n\nThe `advanced.source.batch.size`, `advanced.source.batch.timeout.ms`, and\n`advanced.target.batch.size` properties override their top-level aliases when\nboth forms are present. Change other Flink settings only when instructed by Redis\nsupport. See the [configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processorsadvanced)\nfor all Flink processor properties.\n\n### Choose the Redis data type\n\nSet `target_data_type` to `hash` (the default) or `json`. The `json` option\nrequires JSON support in the target database. A job file can override this\nsetting for its output.\n\n```yaml\nprocessors:\n  type: flink\n  target_data_type: hash\n```\n\nSee [Job files](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples)\nfor the data types available to job outputs.\n\n### Confirm writes reached a replica\n\nUse these properties only when target database replication is enabled and a\nhealthy replica is available:\n\n```yaml\nprocessors:\n  type: flink\n  wait_enabled: true\n  wait_timeout: 1000\n  retry_on_replica_failure: true\n```\n\nFor the Flink processor, the corresponding properties under\n`processors.advanced.target` take priority over these top-level properties.\n\nSee also the\n[RDI configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors)\nfor full details of the other available properties.\n\n## Extended configuration example\n\nThis example combines the commonly used options from this page. Remove properties\nthat you don't need. See the\n[configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference)\nfor every supported property.\n\n```yaml\nsources:\n  mysql:\n    type: cdc\n    logging:\n      level: info\n    connection:\n      type: mysql\n      host: \u003cDB_HOST\u003e # e.g. localhost\n      port: 3306\n      # User and password are injected from the secrets.\n      user: ${SOURCE_DB_USERNAME}\n      password: ${SOURCE_DB_PASSWORD}\n    # Additional properties for the source collector:\n    # List of databases to include (optional).\n    # databases:\n    #   - database1\n    #   - database2\n\n    # List of tables to be synced (optional).\n    # tables:\n    #   If only one database is specified in the databases property above,\n    #   then tables can be defined without the database prefix.\n    #   \u003cDATABASE_NAME\u003e.\u003cTABLE_NAME\u003e:\n    #     List of columns to be synced (optional).\n    #     columns:\n    #       - \u003cCOLUMN_NAME\u003e\n    #       - \u003cCOLUMN_NAME\u003e\n    #     List of columns to be used as keys (optional).\n    #     keys:\n    #       - \u003cCOLUMN_NAME\u003e\n\n    # Example: Sync specific tables.\n    # tables:\n    #   Sync a specific table with all its columns:\n    #   redislabscdc.account: {}\n    #   Sync a specific table with selected columns:\n    #   redislabscdc.emp:\n    #     columns:\n    #       - empno\n    #       - fname\n    #       - lname\n\n    # Advanced collector properties (optional):\n    # advanced:\n    #   Sink collector properties - see the full list at\n    #     https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream\n    #   sink:\n    #     Optional hard limits on memory usage of RDI streams.\n    #     redis.memory.limit.mb: 300\n    #     redis.memory.threshold.percentage: 85\n\n    #     Uncomment for production so RDI Collector will wait on replica\n    #     when writing entries.\n    #     redis.wait.enabled: true\n    #     redis.wait.timeout.ms: 1000\n    #     redis.wait.retry.enabled: true\n    #     redis.wait.retry.delay.ms: 1000\n\n    #   Source specific properties - see the full list at\n    #     https://debezium.io/documentation/reference/stable/connectors/\n    #   source:\n    #     snapshot.mode: initial\n    #     Uncomment if you want a snapshot to include only a subset of the rows\n    #     in a table. This property affects snapshots only.\n    #     snapshot.select.statement.overrides: \u003cDATABASE_NAME\u003e.\u003cTABLE_NAME\u003e\n    #     The specified SELECT statement determines the subset of table rows to\n    #     include in the snapshot.\n    #     snapshot.select.statement.overrides.\u003cDATABASE_NAME\u003e.\u003cTABLE_NAME\u003e: \u003cSELECT_STATEMENT\u003e\n\n    #     Example: Snapshot filtering by order status.\n    #     To include only orders with non-pending status from customers.orders\n    #     table:\n    #     snapshot.select.statement.overrides: customer.orders\n    #     snapshot.select.statement.overrides.customer.orders: SELECT * FROM customers.orders WHERE status != 'pending' ORDER BY order_id DESC\n\n    #   Quarkus framework properties - see the full list at\n    #     https://quarkus.io/guides/all-config\n    #   quarkus:\n    #     banner.enabled: \"false\"\n\n    #   `java_options` (for RDI 1.15.1 and above) controls the JAVA_OPTS environment variable. Use it to modify the default values for\n    #       Java heap size and other Java options for the Debezium server.\n    #   java_options: \"-Xmx2g -Xms512m\"\n\ntargets:\n  # Redis target database connection.\n  # RDI supports one target database. Name it 'target'.\n  target:\n    connection:\n      type: redis\n      # Host of the Redis database to which RDI will\n      # write the processed data.\n      host: \u003cREDIS_TARGET_DB_HOST\u003e # e.g. localhost\n      # Port for the Redis database to which RDI will\n      # write the processed data.\n      port: \u003cREDIS_TARGET_DB_PORT\u003e # e.g. 12000\n      # User of the Redis database to which RDI will write the processed data.\n      # Uncomment if you are not using the default user.\n      # user: ${TARGET_DB_USERNAME}\n      # Password for Redis target database.\n      password: ${TARGET_DB_PASSWORD}\n      # SSL/TLS configuration: Uncomment to enable secure connections.\n      # key: ${TARGET_DB_KEY}\n      # key_password: ${TARGET_DB_KEY_PASSWORD}\n      # cert: ${TARGET_DB_CERT}\n      # cacert: ${TARGET_DB_CACERT}\nprocessors:\n  type: flink\n  # Target data type: hash or json.\n  # target_data_type: hash\n  # Enable merge as the default strategy for writing JSON documents.\n  # json_update_strategy: merge\n  # Confirm that writes reached a target database replica.\n  # wait_enabled: false\n  # wait_timeout: 1000\n  # retry_on_replica_failure: true\n  # Flink processor performance settings.\n  # advanced:\n  #   source:\n  #     batch.size: 2000\n  #     batch.timeout.ms: 100\n  #     discovery.interval.ms: 1000\n  #   target:\n  #     batch.size: 200\n  #     flush.interval.ms: 100\n  #   flink:\n  #     taskmanager.numberOfTaskSlots: 1\n  #     taskmanager.memory.process.size: 2048m\n  #   resources:\n  #     taskManager:\n  #       replicas: 2\n```\n",
  "tags": ["docs","integrate","rs","rdi"],
  "last_updated": "2026-09-19T17:55:58-07:00"
}