{
"id": "config-yaml-reference",
"title": "Redis Data Integration configuration file",
"url": "https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/",
"summary": "Redis Data Integration configuration file reference",
"tags": [
"redis-di"
],
"last_updated": "2026-04-01T08:10:08-05:00",
"page_type": "content",
"content_hash": "bf805ffce7f762e95cd384bf036ee116cfa31e7b45c5b0ce489f09717ddc068b",
"sections": [
{
"id": "overview",
"title": "Overview",
"role": "overview",
"text": "Configuration file for Redis Data Integration (RDI) source collectors and target connections\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ----------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |\n| [**sources**](#sources)
(Source collectors) | `object` | Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
| |\n| [**processors**](#processors)
(Data processing configuration) | `object`, `null` | Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
| |\n| [**targets**](#targets)
(Target connections) | `object` | Configuration for target Redis databases where processed data will be written
| |\n| [**secret\\-providers**](#secret-providers)
(Secret providers) | `object` | Configuration for secret management providers
| |\n\n**Additional Properties:** not allowed \n"
},
{
"id": "sources-source-collectors",
"title": "sources: Source collectors",
"role": "content",
"text": "Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector\n\n**Properties** (key: `.*`)\n\n| Name | Type | Description | Required |\n| ------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------ | -------- |\n| **connection** | | | yes |\n| **type**
(Collector type) | `string` | Type of the source collector.
Default: `\"cdc\"`
Enum: `\"cdc\"`, `\"flink\"`
| yes |\n| **active**
(Collector enabled) | `boolean` | Flag to enable or disable the source collector
Default: `true`
| no |\n| [**logging**](#sourceslogging)
(Logging configuration) | `object` | Logging configuration for the source collector
| no |\n| [**tables**](#sourcestables)
(Tables to capture) | `object` | Defines which tables to capture and how to handle their data
| no |\n| [**schemas**](#sourcesschemas)
(Schema names) | `string[]` | Schema names to capture from the source database (schema.include.list)
| no |\n| [**databases**](#sourcesdatabases)
(Database names) | `string[]` | Database names to capture from the source database (database.include.list)
| no |\n| [**advanced**](#sourcesadvanced)
(Advanced configuration) | `object` | Advanced configuration options for fine-tuning the collector
| no |\n\n"
},
{
"id": "sources-logging-logging-configuration",
"title": "sources\\.logging: Logging configuration",
"role": "content",
"text": "Logging configuration for the source collector\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | -------- |\n| **level**
(Logging level) | `string` | Logging level for the source collector
Default: `\"info\"`
Enum: `\"trace\"`, `\"debug\"`, `\"info\"`, `\"warn\"`, `\"error\"`
| |\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n"
},
{
"id": "sources-tables-tables-to-capture",
"title": "sources\\.tables: Tables to capture",
"role": "content",
"text": "Defines which tables to capture and how to handle their data\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| --------------------------------------------------------------- | ---------------- | ----------- | -------- |\n| [**Additional Properties**](#sourcestablesadditionalproperties) | `object`, `null` | | |\n\n**Minimal Properties:** 1 \n\n\n#### sources\\.tables\\.additionalProperties: object,null\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |\n| **snapshot_sql** | `string` | Custom SQL statement to use for the initial data snapshot, allowing fine-grained control over what data is captured
| |\n| [**columns**](#sourcestablesadditionalpropertiescolumns)
(Columns to capture) | `string[]` | List of specific columns to capture for changes. If not specified, all columns will be captured. Note: This property cannot be used for MongoDB connections
| |\n| [**exclude_columns**](#sourcestablesadditionalpropertiesexclude_columns)
(Columns to exclude) | `string[]` | List of specific columns to exclude from capture. If not specified, no columns will be excluded. Note: This property can only be used for MongoDB connections
| |\n| [**keys**](#sourcestablesadditionalpropertieskeys)
(Message keys) | `string[]` | Optional list of columns to use as a composite unique identifier. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields
| |\n\n**Additional Properties:** not allowed \n\n\n##### sources\\.tables\\.additionalProperties\\.columns\\[\\]: Columns to capture\n\nList of specific columns to capture for changes. If not specified, all columns will be captured. Note: This property cannot be used for MongoDB connections\n\n\n\n##### sources\\.tables\\.additionalProperties\\.exclude_columns\\[\\]: Columns to exclude\n\nList of specific columns to exclude from capture. If not specified, no columns will be excluded. Note: This property can only be used for MongoDB connections\n\n\n\n##### sources\\.tables\\.additionalProperties\\.keys\\[\\]: Message keys\n\nOptional list of columns to use as a composite unique identifier. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields\n\n"
},
{
"id": "sources-schemas-schema-names",
"title": "sources\\.schemas\\[\\]: Schema names",
"role": "content",
"text": "Schema names to capture from the source database (schema.include.list)\n\n"
},
{
"id": "sources-databases-database-names",
"title": "sources\\.databases\\[\\]: Database names",
"role": "content",
"text": "Database names to capture from the source database (database.include.list)\n\n"
},
{
"id": "sources-advanced-advanced-configuration",
"title": "sources\\.advanced: Advanced configuration",
"role": "content",
"text": "Advanced configuration options for fine-tuning the collector\n\n**Properties**\n\n| Name | Type | Description | Required |\n| -------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |\n| [**sink**](#sourcesadvancedsink)
(RDI Collector stream writer configuration) | `object` | Advanced configuration properties for RDI Collector stream writer connection and behaviour. When using collector type 'cdc', see the full list of properties at - https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream . When using a property from that list, remove the `debezium.sink.` prefix. When using collector type 'flink', see the full list of properties at
| |\n| [**source**](#sourcesadvancedsource)
(Advanced source settings) | `object` | Advanced configuration properties for the source database connection and CDC behavior
| |\n| [**quarkus**](#sourcesadvancedquarkus)
(Quarkus runtime settings) | `object` | Advanced configuration properties for the Quarkus runtime environment
| |\n| [**flink**](#sourcesadvancedflink)
(Advanced Flink settings) | `object` | Advanced configuration properties for Flink
| |\n| **java_options**
(Advanced Java options) | `string` | These Java options will be passed to the command line command when launching the source collector
| |\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n\n#### sources\\.advanced\\.sink: RDI Collector stream writer configuration\n\nAdvanced configuration properties for RDI Collector stream writer connection and behaviour. When using collector type 'cdc', see the full list of properties at - https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream . When using a property from that list, remove the `debezium.sink.` prefix. When using collector type 'flink', see the full list of properties at\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n\n\n#### sources\\.advanced\\.source: Advanced source settings\n\nAdvanced configuration properties for the source database connection and CDC behavior\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n\n\n#### sources\\.advanced\\.quarkus: Quarkus runtime settings\n\nAdvanced configuration properties for the Quarkus runtime environment\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n\n\n#### sources\\.advanced\\.flink: Advanced Flink settings\n\nAdvanced configuration properties for Flink\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n"
},
{
"id": "processors-data-processing-configuration",
"title": "processors: Data processing configuration",
"role": "content",
"text": "Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning\n\n**Properties**\n\n| Name | Type | Description | Required |\n| --------------------------------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |\n| **type**
(Processor type) | `string` | Processor type, either 'classic' or 'flink'
Default: `\"classic\"`
Enum: `\"classic\"`
| |\n| **on_failed_retry_interval**
(Retry interval on failure) | `integer`, `string` | Number of seconds to wait before retrying a failed operation
Default: `5`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **read_batch_size** | `integer`, `string` | Maximum number of records to process in a single batch
Default: `2000`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **read_batch_timeout_ms**
(Read batch timeout) | `integer` | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Minimum: `1`
| |\n| **enable_async_processing** | `boolean` | Enable async processing to improve throughput
Default: `true`
| |\n| **batch_queue_size** | `integer` | Maximum number of batches to queue for processing
Default: `3`
Minimum: `1`
| |\n| **ack_queue_size** | `integer` | Maximum number of batches to queue for asynchronous acknowledgement
Default: `10`
Minimum: `1`
| |\n| **dedup**
(Enable deduplication) | `boolean` | Enable the deduplication mechanism to handle duplicate records
Default: `false`
| |\n| **dedup_max_size**
(Deduplication set size) | `integer` | Maximum number of entries to store in the deduplication set
Default: `1024`
Minimum: `1`
| |\n| **dedup_strategy**
(Deduplication strategy) | `string` | (DEPRECATED)
Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration.
Default: `\"ignore\"`
Enum: `\"reject\"`, `\"ignore\"`
| |\n| **duration**
(Batch duration limit) | `integer`, `string` | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **write_batch_size** | `integer`, `string` | Maximum number of records to write to target Redis database in a single batch
Default: `200`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **error_handling**
(Error handling strategy) | `string` | Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue
Default: `\"dlq\"`
Pattern: `^\\${.*}$\\|ignore\\|dlq`
| |\n| **dlq_max_messages**
(DLQ message limit) | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **target_data_type**
(Target Redis data type) | `string` | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)
Default: `\"hash\"`
Pattern: `^\\${.*}$\\|hash\\|json`
| |\n| **json_update_strategy** | `string` | Strategy for updating JSON data in Redis: replace to overwrite the entire JSON object, merge to merge new data with existing JSON object
Default: `\"replace\"`
Pattern: `^\\${.*}$\\|replace\\|merge`
| |\n| **initial_sync_processes** | `integer`, `string` | Number of parallel processes for performing initial data synchronization
Default: `4`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `32`
| |\n| **idle_sleep_time_ms**
(Idle sleep interval) | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle
Default: `200`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
| |\n| **idle_streams_check_interval_ms**
(Idle streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
| |\n| **busy_streams_check_interval_ms**
(Busy streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is busy
Default: `5000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
| |\n| **wait_enabled**
(Enable replica wait) | `boolean` | Enable verification that data has been written to replica shards of the target database
Default: `false`
| |\n| **wait_timeout**
(Replica wait timeout) | `integer`, `string` | Maximum time in milliseconds to wait for replica write verification of the target database
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **retry_max_attempts**
(Maximum retry attempts) | `integer`, `string` | Maximum number of attempts for failed operations
Default: `5`
Pattern: `^\\${.*}$`
Minimum: `1`
| |\n| **retry_initial_delay_ms**
(Initial retry delay) | `integer`, `string` | Initial delay in milliseconds before retrying a failed operation
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
| |\n| **retry_max_delay_ms**
(Maximum retry delay) | `integer`, `string` | Maximum delay in milliseconds between retry attempts
Default: `10000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
| |\n| **retry_on_replica_failure** | `boolean` | Continue retrying writes until successful replication to replica shards is confirmed
Default: `true`
| |\n| [**logging**](#processorslogging)
(Logging configuration) | `object` | Logging configuration for the processor
| |\n| **use_native_json_merge**
(Use native JSON merge from RedisJSON module) | `boolean` | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:
**Previous behavior (Lua merge)**: When merging `{\"field1\": \"value1\", \"field2\": \"value2\"}` with `{\"field2\": null, \"field3\": \"value3\"}`, the result was `{\"field1\": \"value1\", \"field2\": null, \"field3\": \"value3\"}` (null value is preserved)
**New behavior (JSON.MERGE)**: The same merge produces `{\"field1\": \"value1\", \"field3\": \"value3\"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))
**Note**: The native `JSON.MERGE` command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI will automatically fall back to using Lua-based merge operations regardless of this setting.
**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.
Default: `true`
| |\n| [**advanced**](#processorsadvanced)
(Advanced configuration) | `object` | Advanced configuration options for fine-tuning the processor
| |\n\n**Additional Properties:** not allowed \n"
},
{
"id": "processors-logging-logging-configuration",
"title": "processors\\.logging: Logging configuration",
"role": "content",
"text": "Logging configuration for the processor\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ | -------- |\n| **level**
(Logging level) | `string` | Logging level for the processor
Default: `\"info\"`
Enum: `\"trace\"`, `\"debug\"`, `\"info\"`, `\"warn\"`, `\"error\"`
| |\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n"
},
{
"id": "processors-advanced-advanced-configuration",
"title": "processors\\.advanced: Advanced configuration",
"role": "content",
"text": "Advanced configuration options for fine-tuning the processor\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ------------------------------------------------------------------------------- | -------- | -------------------------------------------------------- | -------- |\n| [**source**](#processorsadvancedsource)
(Advanced source settings) | `object` | Advanced configuration properties for the source
| |\n| [**sink**](#processorsadvancedsink)
(Advanced sink settings) | `object` | Advanced configuration properties for the sink
| |\n| [**processor**](#processorsadvancedprocessor)
(Advanced processor settings) | `object` | Advanced configuration properties for the processor
| |\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n\n#### processors\\.advanced\\.source: Advanced source settings\n\nAdvanced configuration properties for the source\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n\n\n#### processors\\.advanced\\.sink: Advanced sink settings\n\nAdvanced configuration properties for the sink\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n\n\n#### processors\\.advanced\\.processor: Advanced processor settings\n\nAdvanced configuration properties for the processor\n\n**Additional Properties**\n\n| Name | Type | Description | Required |\n| ------------------------- | ----------------------------- | ----------- | -------- |\n| **Additional Properties** | `string`, `number`, `boolean` | | |\n\n**Minimal Properties:** 1 \n"
},
{
"id": "targets-target-connections",
"title": "targets: Target connections",
"role": "content",
"text": "Configuration for target Redis databases where processed data will be written\n\n**Properties (Pattern)**\n\n| Name | Type | Description | Required |\n| -------- | ---- | ----------- | -------- |\n| **\\.\\*** | | | |\n\n"
},
{
"id": "secret-providers-secret-providers",
"title": "secret\\-providers: Secret providers",
"role": "content",
"text": "Configuration for secret management providers\n\n**Properties** (key: `.*`)\n\n| Name | Type | Description | Required |\n| ----------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- | -------- |\n| **type**
(Provider type) | `string` | Type of secret provider service
Enum: `\"aws\"`, `\"vault\"`
| yes |\n| [**parameters**](#secret-providersparameters)
(Provider parameters) | `object` | Configuration parameters for the secret provider
| yes |\n\n"
},
{
"id": "secret-providers-parameters-provider-parameters",
"title": "secret\\-providers\\.parameters: Provider parameters",
"role": "content",
"text": "Configuration parameters for the secret provider\n\n**Properties**\n\n| Name | Type | Description | Required |\n| ----------------------------------------------------------------------------- | ---------- | ------------------------------------------------------ | -------- |\n| [**objects**](#secret-providersparametersobjects)
(Secrets objects array) | `object[]` | List of secret objects to fetch from the provider
| yes |\n\n**Example**\n\n[code example]\n\n\n\n#### secret\\-providers\\.parameters\\.objects\\[\\]: Secrets objects array\n\nList of secret objects to fetch from the provider\n\n**Items: Secret object**\n\n**No properties.**\n\n**Example**\n\n[code example]"
}
],
"examples": [
{
"id": "sources-logging-logging-configuration-ex0",
"language": "yaml",
"code": "level: info",
"section_id": "sources-logging-logging-configuration"
},
{
"id": "sources-advanced-advanced-configuration-ex0",
"language": "yaml",
"code": "sink: {}\nsource: {}\nquarkus: {}\nflink: {}",
"section_id": "sources-advanced-advanced-configuration"
},
{
"id": "processors-logging-logging-configuration-ex0",
"language": "yaml",
"code": "level: info",
"section_id": "processors-logging-logging-configuration"
},
{
"id": "processors-advanced-advanced-configuration-ex0",
"language": "yaml",
"code": "source: {}\nsink: {}\nprocessor: {}",
"section_id": "processors-advanced-advanced-configuration"
},
{
"id": "secret-providers-parameters-provider-parameters-ex0",
"language": "yaml",
"code": "objects:\n - {}",
"section_id": "secret-providers-parameters-provider-parameters"
},
{
"id": "secret-providers-parameters-provider-parameters-ex1",
"language": "yaml",
"code": "- {}",
"section_id": "secret-providers-parameters-provider-parameters"
}
]
}