summary: Redis Data Integration configuration file reference

title: Overview

text:
Configuration file for Redis Data Integration (RDI) source collectors and target connections

**Properties**

| Name                                                              | Type             | Description                                                                                | Required |
| ----------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| [**sources**](#sources)
(Source collectors)                   | `object`         | Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
 |          |
| [**processors**](#processors)
(Data processing configuration) | `object`, `null` | Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
                                             |          |
| [**targets**](#targets)
(Target connections)                  | `object`         | Configuration for target Redis databases where processed data will be written
                                                                                |          |
| [**secret\-providers**](#secret-providers)
(Secret providers) | `object`         | Configuration for secret management providers
                                                                                |          |

**Additional Properties:** not allowed

title: sources: Source collectors

text:
Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector

**Properties** (key: `.*`)

| Name                                                          | Type       | Description                                                                          | Required |
| ------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------ | -------- |
| **connection**                                                |            |                                                                                | yes      |
| **type**
(Collector type)                                 | `string`   | Type of the source collector.
Default: `"cdc"`
Enum: `"cdc"`, `"flink"`
 | yes      |
| **active**
(Collector enabled)                            | `boolean`  | Flag to enable or disable the source collector
Default: `true`
              | no       |
| [**logging**](#sourceslogging)
(Logging configuration)    | `object`   | Logging configuration for the source collector
                                  | no       |
| [**tables**](#sourcestables)
(Tables to capture)          | `object`   | Defines which tables to capture and how to handle their data
                    | no       |
| [**schemas**](#sourcesschemas)
(Schema names)             | `string[]` | Schema names to capture from the source database (schema.include.list)
          | no       |
| [**databases**](#sourcesdatabases)
(Database names)       | `string[]` | Database names to capture from the source database (database.include.list)
      | no       |
| [**advanced**](#sourcesadvanced)
(Advanced configuration) | `object`   | Advanced configuration options for fine-tuning the collector
                    | no       |

title: sources\.logging: Logging configuration

text:
Logging configuration for the source collector

**Properties**

| Name                          | Type     | Description                                                                                | Required |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | -------- |
| **level**
(Logging level) | `string` | Logging level for the source collector
Default: `"info"`
Enum: `"trace"`, `"debug"`, `"info"`, `"warn"`, `"error"`
 |          |

**Additional Properties:** not allowed
**Example**

[code example]

title: sources\.tables: Tables to capture

text:
Defines which tables to capture and how to handle their data

**Additional Properties**

| Name                                                            | Type             | Description | Required |
| --------------------------------------------------------------- | ---------------- | ----------- | -------- |
| [**Additional Properties**](#sourcestablesadditionalproperties) | `object`, `null` |             |          |

**Minimal Properties:** 1

#### sources\.tables\.additionalProperties: object,null

**Properties**

| Name                                                                                | Type       | Description                                                                                | Required |
| ------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| **snapshot_sql**                                                                                | `string`   | Custom SQL statement to use for the initial data snapshot, allowing fine-grained control over what data is captured
                                                               |          |
| [**columns**](#sourcestablesadditionalpropertiescolumns)
(Columns to capture)                 | `string[]` | List of specific columns to capture for changes. If not specified, all columns will be captured. Note: This property cannot be used for MongoDB connections
                       |          |
| [**exclude_columns**](#sourcestablesadditionalpropertiesexclude_columns)
(Columns to exclude) | `string[]` | List of specific columns to exclude from capture. If not specified, no columns will be excluded. Note: This property can only be used for MongoDB connections
                     |          |
| [**keys**](#sourcestablesadditionalpropertieskeys)
(Message keys)                             | `string[]` | Optional list of columns to use as a composite unique identifier. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields
 |          |

**Additional Properties:** not allowed

##### sources\.tables\.additionalProperties\.columns\[\]: Columns to capture

List of specific columns to capture for changes. If not specified, all columns will be captured. Note: This property cannot be used for MongoDB connections

##### sources\.tables\.additionalProperties\.exclude_columns\[\]: Columns to exclude

List of specific columns to exclude from capture. If not specified, no columns will be excluded. Note: This property can only be used for MongoDB connections

##### sources\.tables\.additionalProperties\.keys\[\]: Message keys

Optional list of columns to use as a composite unique identifier. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields

title: sources\.schemas\[\]: Schema names

title: sources\.databases\[\]: Database names

title: sources\.advanced: Advanced configuration

text:
Advanced configuration options for fine-tuning the collector

**Properties**

| Name                                                                             | Type     | Description                                                                                | Required |
| -------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| [**sink**](#sourcesadvancedsink)
(RDI Collector stream writer configuration) | `object` | Advanced configuration properties for RDI Collector stream writer connection and behaviour. When using collector type 'cdc', see the full list of properties at - https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream . When using a property from that list, remove the `debezium.sink.` prefix. When using collector type 'flink', see the full list of properties at
 |          |
| [**source**](#sourcesadvancedsource)
(Advanced source settings)              | `object` | Advanced configuration properties for the source database connection and CDC behavior
                                                                                |          |
| [**quarkus**](#sourcesadvancedquarkus)
(Quarkus runtime settings)            | `object` | Advanced configuration properties for the Quarkus runtime environment
                                                                                |          |
| [**flink**](#sourcesadvancedflink)
(Advanced Flink settings)                 | `object` | Advanced configuration properties for Flink
                                                                                |          |
| **java_options**
(Advanced Java options)                                     | `string` | These Java options will be passed to the command line command when launching the source collector
                                                                                |          |

**Additional Properties:** not allowed
**Minimal Properties:** 1
**Example**

[code example]

#### sources\.advanced\.sink: RDI Collector stream writer configuration

Advanced configuration properties for RDI Collector stream writer connection and behaviour. When using collector type 'cdc', see the full list of properties at - https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream . When using a property from that list, remove the `debezium.sink.` prefix. When using collector type 'flink', see the full list of properties at

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

#### sources\.advanced\.source: Advanced source settings

Advanced configuration properties for the source database connection and CDC behavior

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

#### sources\.advanced\.quarkus: Quarkus runtime settings

Advanced configuration properties for the Quarkus runtime environment

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

#### sources\.advanced\.flink: Advanced Flink settings

Advanced configuration properties for Flink

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

title: processors: Data processing configuration

text:
Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning

**Properties**

| Name                                                                        | Type                | Description                                                                                | Required |
| --------------------------------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| **type**
(Processor type)                                               | `string`            | Processor type, either 'classic' or 'flink'
Default: `"classic"`
Enum: `"classic"`
                                                                                |          |
| **on_failed_retry_interval**
(Retry interval on failure)                | `integer`, `string` | Number of seconds to wait before retrying a failed operation
Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **read_batch_size**                                                         | `integer`, `string` | Maximum number of records to process in a single batch
Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **read_batch_timeout_ms**
(Read batch timeout)                          | `integer`           | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Minimum: `1`
                                                                                |          |
| **enable_async_processing**                                                 | `boolean`           | Enable async processing to improve throughput
Default: `true`
                                                                                |          |
| **batch_queue_size**                                                        | `integer`           | Maximum number of batches to queue for processing
Default: `3`
Minimum: `1`
                                                                                |          |
| **ack_queue_size**                                                          | `integer`           | Maximum number of batches to queue for asynchronous acknowledgement
Default: `10`
Minimum: `1`
                                                                                |          |
| **dedup**
(Enable deduplication)                                        | `boolean`           | Enable the deduplication mechanism to handle duplicate records
Default: `false`
                                                                                |          |
| **dedup_max_size**
(Deduplication set size)                             | `integer`           | Maximum number of entries to store in the deduplication set
Default: `1024`
Minimum: `1`
                                                                                |          |
| **dedup_strategy**
(Deduplication strategy)                             | `string`            | (DEPRECATED)
Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration.
Default: `"ignore"`
Enum: `"reject"`, `"ignore"`
                                                                                |          |
| **duration**
(Batch duration limit)                                     | `integer`, `string` | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **write_batch_size**                                                        | `integer`, `string` | Maximum number of records to write to target Redis database in a single batch
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **error_handling**
(Error handling strategy)                            | `string`            | Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue
Default: `"dlq"`
Pattern: `^\${.*}$\|ignore\|dlq`
                                                                                |          |
| **dlq_max_messages**
(DLQ message limit)                                | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **target_data_type**
(Target Redis data type)                           | `string`            | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)
Default: `"hash"`
Pattern: `^\${.*}$\|hash\|json`
                                                                                |          |
| **json_update_strategy**                                                    | `string`            | Strategy for updating JSON data in Redis: replace to overwrite the entire JSON object, merge to merge new data with existing JSON object
Default: `"replace"`
Pattern: `^\${.*}$\|replace\|merge`
                                                                                |          |
| **initial_sync_processes**                                                  | `integer`, `string` | Number of parallel processes for performing initial data synchronization
Default: `4`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `32`
                                                                                |          |
| **idle_sleep_time_ms**
(Idle sleep interval)                            | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
                                                                                |          |
| **idle_streams_check_interval_ms**
(Idle streams check interval)        | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
                                                                                |          |
| **busy_streams_check_interval_ms**
(Busy streams check interval)        | `integer`, `string` | Time in milliseconds between checking for new streams when processor is busy
Default: `5000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
                                                                                |          |
| **wait_enabled**
(Enable replica wait)                                  | `boolean`           | Enable verification that data has been written to replica shards of the target database
Default: `false`
                                                                                |          |
| **wait_timeout**
(Replica wait timeout)                                 | `integer`, `string` | Maximum time in milliseconds to wait for replica write verification of the target database
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **retry_max_attempts**
(Maximum retry attempts)                         | `integer`, `string` | Maximum number of attempts for failed operations
Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **retry_initial_delay_ms**
(Initial retry delay)                        | `integer`, `string` | Initial delay in milliseconds before retrying a failed operation
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
                                                                                |          |
| **retry_max_delay_ms**
(Maximum retry delay)                            | `integer`, `string` | Maximum delay in milliseconds between retry attempts
Default: `10000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
                                                                                |          |
| **retry_on_replica_failure**                                                | `boolean`           | Continue retrying writes until successful replication to replica shards is confirmed
Default: `true`
                                                                                |          |
| [**logging**](#processorslogging)
(Logging configuration)               | `object`            | Logging configuration for the processor
                                                                                |          |
| **use_native_json_merge**
(Use native JSON merge from RedisJSON module) | `boolean`           | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:

**Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)

**New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))

**Note**: The native `JSON.MERGE` command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI will automatically fall back to using Lua-based merge operations regardless of this setting.

**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.
Default: `true`
 |          |
| [**advanced**](#processorsadvanced)
(Advanced configuration)            | `object`            | Advanced configuration options for fine-tuning the processor
                                                                                |          |

**Additional Properties:** not allowed

title: processors\.logging: Logging configuration

text:
Logging configuration for the processor

**Properties**

| Name                          | Type     | Description                                                                                | Required |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ | -------- |
| **level**
(Logging level) | `string` | Logging level for the processor
Default: `"info"`
Enum: `"trace"`, `"debug"`, `"info"`, `"warn"`, `"error"`
 |          |

**Additional Properties:** not allowed
**Example**

[code example]

title: processors\.advanced: Advanced configuration

text:
Advanced configuration options for fine-tuning the processor

**Properties**

| Name                                                                            | Type     | Description                                              | Required |
| ------------------------------------------------------------------------------- | -------- | -------------------------------------------------------- | -------- |
| [**source**](#processorsadvancedsource)
(Advanced source settings)          | `object` | Advanced configuration properties for the source
    |          |
| [**sink**](#processorsadvancedsink)
(Advanced sink settings)                | `object` | Advanced configuration properties for the sink
      |          |
| [**processor**](#processorsadvancedprocessor)
(Advanced processor settings) | `object` | Advanced configuration properties for the processor
 |          |

**Additional Properties:** not allowed
**Minimal Properties:** 1
**Example**

[code example]

#### processors\.advanced\.source: Advanced source settings

Advanced configuration properties for the source

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

#### processors\.advanced\.sink: Advanced sink settings

Advanced configuration properties for the sink

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

#### processors\.advanced\.processor: Advanced processor settings

Advanced configuration properties for the processor

**Additional Properties**

| Name                      | Type                          | Description | Required |
| ------------------------- | ----------------------------- | ----------- | -------- |
| **Additional Properties** | `string`, `number`, `boolean` |             |          |

**Minimal Properties:** 1

title: targets: Target connections

text:
Configuration for target Redis databases where processed data will be written

**Properties (Pattern)**

| Name     | Type | Description | Required |
| -------- | ---- | ----------- | -------- |
| **\.\*** |      |             |          |

title: secret\-providers: Secret providers

text:
Configuration for secret management providers

**Properties** (key: `.*`)

| Name                                                                    | Type     | Description                                                       | Required |
| ----------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- | -------- |
| **type**
(Provider type)                                            | `string` | Type of secret provider service
Enum: `"aws"`, `"vault"`
 | yes      |
| [**parameters**](#secret-providersparameters)
(Provider parameters) | `object` | Configuration parameters for the secret provider
             | yes      |

title: secret\-providers\.parameters: Provider parameters

text:
Configuration parameters for the secret provider

**Properties**

| Name                                                                          | Type       | Description                                            | Required |
| ----------------------------------------------------------------------------- | ---------- | ------------------------------------------------------ | -------- |
| [**objects**](#secret-providersparametersobjects)
(Secrets objects array) | `object[]` | List of secret objects to fetch from the provider
 | yes      |

**Example**

[code example]

#### secret\-providers\.parameters\.objects\[\]: Secrets objects array

List of secret objects to fetch from the provider

**Items: Secret object**

**No properties.**

**Example**

[code example]

code: level: info

section_id: sources-logging-logging-configuration

code:
sink: {}
source: {}
quarkus: {}
flink: {}

section_id: sources-advanced-advanced-configuration

code:
source: {}
sink: {}
processor: {}

code:
objects:
  - {}


title: add_field

summary: Add fields to a record

page_type: content

title: Overview

role: overview

text:
Add fields to a record

**Option 1 (alternative):**
Add multiple fields

**Properties**

| Name                         | Type       | Description | Required |
| ---------------------------- | ---------- | ----------- | -------- |
| [**fields**](#option1fields) | `object[]` | Fields
 | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

**Option 2 (alternative):**
Add one field

**Properties**

| Name           | Type     | Description                                   | Required |
| -------------- | -------- | --------------------------------------------- | -------- |
| **field**      | `string` | Field
                                    | yes      |
| **expression** | `string` | Expression
                               | yes      |
| **language**   | `string` | Language
Enum: `"jmespath"`, `"sql"`
 | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

title: Option 1: fields\[\]: array

role: parameters

text:
Fields

**Items**

**Item Properties**

| Name           | Type     | Description                                   | Required |
| -------------- | -------- | --------------------------------------------- | -------- |
| **field**      | `string` | Field
                                    | yes      |
| **expression** | `string` | Expression
                               | yes      |
| **language**   | `string` | Language
Enum: `"jmespath"`, `"sql"`
 | yes      |

**Item Additional Properties:** not allowed

**Example**

[code example]

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: add_field
    with:
      fields:
        - field: name.full_name
          language: jmespath
          expression: concat([name.fname, ' ', name.lname])
        - field: name.fname_upper
          language: jmespath
          expression: upper(name.fname)

section_id: overview

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: add_field
    with:
      field: country
      language: sql
      expression: country_code || ' - ' || UPPER(country_name)


title: filter

summary: Filter records

title: Content

role: content

text:
Filter records

**Properties**

| Name           | Type     | Description                                   | Required |
| -------------- | -------- | --------------------------------------------- | -------- |
| **expression** | `string` | Expression
                               | yes      |
| **language**   | `string` | Language
Enum: `"jmespath"`, `"sql"`
 | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: filter
    with:
      language: sql
      expression: age>20

section_id: content


title: Data transformation reference

summary: View reference material for RDI data transformations

summary: Add fields to a record

title: add_field

summary: Filter records

title: filter

summary: Map a record into a new output based on expressions

title: map

summary: Lookup data from Redis using the given command and key

title: redis.lookup

summary: Remove fields

title: remove_field

summary: Rename fields. All other fields remain unchanged.

title: rename_field

page_type: index


title: redis.lookup

title: Overview

text:
**Properties**

| Name              | Type       | Description                                   | Required |
| ----------------- | ---------- | --------------------------------------------- | -------- |
| **connection**    | `string`   | Connection name                               | yes      |
| **cmd**           | `string`   | The command to execute                        | yes      |
| [**args**](#args) | `string[]` | Redis command arguments                       | yes      |
| **language**      | `string`   | Language
Enum: `"jmespath"`, `"sql"`
 | yes      |
| **field**         | `string`   | The target field to write the result to
  | yes      |

**Additional Properties:** not allowed

title: args\[\]: Redis command arguments {#args}

text:
The list of expressions that produce arguments.

**Items**

**Item Type:** `string`


title: map

title: Overview

text:
Map a record into a new output based on expressions

**Properties**

| Name                          | Type               | Description                                   | Required |
| ----------------------------- | ------------------ | --------------------------------------------- | -------- |
| [**expression**](#expression) | `object`, `string` | Expression
                               | yes      |
| **language**                  | `string`           | Language
Enum: `"jmespath"`, `"sql"`
 | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

**Example**

[code example]

title: expression: object

text:
Expression

**No properties.**

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: map
    with:
      expression:
        first_name: first_name
        last_name: last_name
        greeting: >-
          'Hello ' || CASE WHEN gender = 'F' THEN 'Ms.' WHEN gender = 'M' THEN 'Mr.'
          ELSE 'N/A' END || ' ' || full_name
        country: country
        full_name: full_name
      language: sql

code:
source:
  table: customer
transform:
  - uses: map
    with:
      expression: |
        {
          "CustomerId": customer_id,
          "FirstName": first_name,
          "LastName": last_name,
          "Company": company,
          "Location":
          {
            "Street": address,
            "City": city,
            "State": state,
            "Country": country,
            "PostalCode": postal_code
          },
          "Phone": phone,
          "Fax": fax,
          "Email": email
        }
      language: jmespath


title: remove_field

summary: Remove fields

title: Overview

text:
Remove fields

**Option 1 (alternative):**
Remove multiple fields

**Properties**

| Name                         | Type       | Description | Required |
| ---------------------------- | ---------- | ----------- | -------- |
| [**fields**](#option1fields) | `object[]` | Fields
 | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

**Option 2 (alternative):**
Remove one field

**Properties**

| Name      | Type     | Description | Required |
| --------- | -------- | ----------- | -------- |
| **field** | `string` | Field
  | yes      |

**Additional Properties:** not allowed
**Example**

[code example]

title: Option 1: fields\[\]: array

text:
Fields

**Items**

**Item Properties**

| Name      | Type     | Description | Required |
| --------- | -------- | ----------- | -------- |
| **field** | `string` | Field<br/>  | yes      |

**Item Additional Properties:** not allowed

**Example**

[code example]

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: remove_field
    with:
      fields:
        - field: credit_card
        - field: name.mname

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: remove_field
    with:
      field: credit_card


title: rename_field

title: Overview

text:
Rename fields. All other fields remain unchanged.

**Option 1 (alternative):**
Rename multiple fields

**Properties**

| Name                         | Type       | Description | Required |
| ---------------------------- | ---------- | ----------- | -------- |
| [**fields**](#option1fields) | `object[]` | Fields
 | yes      |

**Additional Properties:** not allowed
**Example**

[code example]

**Option 2 (alternative):**
Rename one field

**Properties**

| Name           | Type     | Description     | Required |
| -------------- | -------- | --------------- | -------- |
| **from_field** | `string` | From field
 | yes      |
| **to_field**   | `string` | To field
   | yes      |

**Additional Properties:** not allowed

**Example**

[code example]

title: Option 1: fields\[\]: array

text:
Fields

**Items**

**Item Properties**

| Name           | Type     | Description     | Required |
| -------------- | -------- | --------------- | -------- |
| **from_field** | `string` | From field
 | yes      |
| **to_field**   | `string` | To field
   | yes      |

**Item Additional Properties:** not allowed

**Example**

[code example]

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: rename_field
    with:
      fields:
        - from_field: name.lname
          to_field: name.last_name
        - from_field: name.fname
          to_field: name.first_name

code:
source:
  schema: dbo
  table: emp
transform:
  - uses: rename_field
    with:
      from_field: name.lname
      to_field: name.last_name


title: Reference

summary: View reference material for Redis Data Integration

title: Redis Data Integration configuration file

summary: JMESPath custom function reference

title: JMESPath custom functions

summary: Reference for the RDI CLI commands

title: CLI reference

title: Data transformation reference

title: Redis Data Integration API


title: JMESPath custom functions

summary: JMESPath custom function reference

title: Content

text:
See also the [JMESPath functions proposal](https://jmespath.org/proposals/functions.html)
for a full description of the function syntax and a list of built-in functions.

| Function             | Description                                                                                | Example                                                                                | Comments                                                                                |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base64_decode`      | Decodes a base64(RFC 4648) encoded string                                                                                | Input: `{"encoded": "SGVsbG8gV29ybGQh"}`
 Expression: `base64_decode(encoded)`
 Output: `Hello World!`                                                                                |                                                                                |
| `capitalize`         | Capitalizes all the words in the string                                                                                | Input: `{"name": "john doe"}`
 Expression: `capitalize(name)`
 Output: `John Doe`                                                                                |                                                                                |
| `concat`             | Concatenates an array of variables or literals                                                                                | Input: `{"fname": "john", "lname": "doe"}`
 Expression: `concat([fname, ' ' ,lname])`
 Output: `john doe`                                                                                | This is equivalent to the more verbose built-in expression: `' '.join([fname,lname])`                                                                            |
| `filter_entries`     | Filters entries in a dictionary (object) using the given JMESPath predicate                                                         | Input: `{ "name": "John", "age": 30, "country": "US", "score": 15}`
 Expression: `` filter_entries(@, `key == 'name' \|\| key == 'age'`)``
 Output:`{"name": "John", "age": 30 }`             |                                                                                |
| `from_entries`       | Converts an array of objects with `key` and `value` properties into a single object                                                 | Input: `[{"key": "name", "value": "John"}, {"key": "age", "value": 30}, {"key": "city", "value": null}]`
 Expression: `from_entries(@)`
 Output: `{"name": "John", "age": 30, "city": null}` |                                                                                |
| `hash`               | Calculates a hash using the `hash_name` hash function and returns its hexadecimal representation                                    | Input: `{"some_str": "some_value"}`
 Expression: `hash(some_str, `sha1`)`
 Output: `8c818171573b03feeae08b0b4ffeb6999e3afc05`                                                        | Supported algorithms: sha1 (default), sha256, md5, sha384, sha3_384, blake2b, sha512, sha3_224, sha224, sha3_256, sha3_512, blake2s                              |
| `in`                 | Checks if an element matches any value in a list of values                                                                          | Input: `{"el": "b"}`
 Expression: `in(el, `["a", "b", "c"]`)`
 Output: `True`                                                                                |                                                                                |
| `left`               | Returns a specified number of characters from the start of a given text string                                                      | Input: `{"greeting": "hello world!"}`
 Expression: `left(greeting, `5`)`
 Output: `hello`                                                                                |                                                                                |
| `lower`              | Converts all uppercase characters in a string into lowercase characters                                                             | Input: `{"fname": "John"}`
 Expression: `lower(fname)`
 Output: `john`                                                                                |                                                                                |
| `mid`                | Returns a specified number of characters from the middle of a given text string                                                     | Input: `{"greeting": "hello world!"}`
 Expression: `mid(greeting, `4`, `3`)`
 Output: `o w`                                                                                |                                                                                |
| `json_parse`         | Returns parsed object from the given JSON string                                                                                | Input: `{"data": '{"greeting": "hello world!"}'}`
 Expression: `json_parse(data)`
 Output: `{"greeting": "hello world!"}`                                                                    |                                                                                |
| `regex_replace`      | Replaces a string that matches a regular expression                                                                                | Input: `{"text": "Banana Bannnana"}`
 Expression: `regex_replace(text, 'Ban\w+', 'Apple Apple')`
 Output: `Apple Apple`                                                                      |                                                                                |
| `replace`            | Replaces all the occurrences of a substring with a new one                                                                          | Input: `{"sentence": "one four three four!"}`
 Expression: `replace(sentence, 'four', 'two')`
 Output: `one two three two!`                                                                  |                                                                                |
| `right`              | Returns a specified number of characters from the end of a given text string                                                        | Input: `{"greeting": "hello world!"}`
 Expression: `right(greeting, `6`)`
 Output: `world!`                                                                                |                                                                                |
| `split`              | Splits a string into a list of strings after breaking the given string by the specified delimiter (comma by default)                | Input: `{"departments": "finance,hr,r&d"}`
 Expression: `split(departments)`
 Output: `['finance', 'hr', 'r&d']`                                                                             | Default delimiter is comma - a different delimiter can be passed to the function as the second argument, for example: `split(departments, ';')`                  |
| `time_delta_days`    | Returns the number of days between a given `dt` and now (positive) or the number of days that have passed from now (negative)       | Input: `{"dt": '2021-10-06T18:56:16.701670+00:00'}`
 Expression: `time_delta_days(dt)`
 Output: `365`                                                                                | If `dt` is a string, ISO datetime (2011-11-04T00:05:23+04:00, for example) is assumed. If `dt` is a number, Unix timestamp (1320365123, for example) is assumed. |
| `time_delta_seconds` | Returns the number of seconds between a given `dt` and now (positive) or the number of seconds that have passed from now (negative) | Input: `{"dt": '2021-10-06T18:56:16.701670+00:00'}`
 Expression: `time_delta_days(dt)`
 Output: `31557600`                                                                                | If `dt` is a string, ISO datetime (2011-11-04T00:05:23+04:00, for example) is assumed. If `dt` is a number, Unix timestamp (1320365123, for example) is assumed. |
| `to_entries`         | Converts a given object into an array of objects with `key` and `value` properties                                                  | Input: `{"name": "John", "age": 30, "city": null}`
 Expression: `to_entries(@)`
 Output: `[{"key": "name", "value": "John"}, {"key": "age", "value": 30}, {"key": "city", "value": null}]`   |                                                                                |
| `upper`              | Converts all lowercase characters in a string into uppercase characters                                                             | Input: `{"fname": "john"}`
 Expression: `upper(fname)`
 Output: `JOHN`                                                                                |                                                                                |
| `uuid`               | Generates a random UUID4 and returns it as a string in standard format                                                              | Input: None
 Expression: `uuid()`
 Output: `3264b35c-ff5d-44a8-8bc7-9be409dac2b7`                                                                                |                                                                                |
| `xml_to_dict`        | Converts an XML string to a dictionary                                                                                | Input: `{"xml": "<root><name>John</name><age>30</age></root>"}`
 Expression: `xml_to_dict(xml)`
 Output: `{"root": {"name": "John", "age": "30"}}`                                            | Returns `null` if the input is `null`; returns an empty string if the input is an empty string                                                                   |


title: Release notes

summary: Maintenance release focusing on allowing the use of RDI with Redis Enterprise databases that have AOF disabled.

title: Redis Data Integration release notes 1.16.2 (March 2026)

summary: Maintenance release focusing on TLS certificate handling fixes and API v2 pipeline patching. Includes security hardening for the collector API through dependency upgrades (Spring Boot, Netty, Kafka clients, MySQL connector, Logback, Jetty, Spring Security, Spring Framework).

title: Redis Data Integration release notes 1.16.1 (February 2026)

summary: Enhanced disaster recovery for Cloud Spanner with Redis Active-Active, plus better performance and resilience in the Spanner collector. New RDI API v2 dry-run, info, and metrics endpoints for safer rollouts and improved observability. Improved deployment, installation, and troubleshooting, including disk space preflight checks, and broader stability fixes.

title: Redis Data Integration release notes 1.16.0 (January 2026)

summary: Fixes support package generation and masking to simplify troubleshooting and log collection. Improves processor throughput during initial sync via parallel processing and new stream tuning parameters. Adds an advanced configuration option to pass custom Java options to the Debezium collector.

title: Redis Data Integration release notes 1.15.1 (November 2025)

summary: Flink collector for Spanner enabled by default for improved user experience. Enhanced high availability with configurable leader election and standby mode. Support for sharded RDI Redis databases. Improved configuration validation and monitoring capabilities. Better resource management and security enhancements.

title: Redis Data Integration release notes 1.15.0 (October 2025)

summary: New RDI API v2 with enhanced pipeline management. Improved Oracle RAC support with configuration scaffolding. Enhanced metrics and monitoring capabilities. Better TLS/mTLS support across components.

title: Redis Data Integration release notes 1.14.1 (August 2025)

title: Redis Data Integration release notes 1.14.0 (August 2025)

summary: Fixed operator excessive CPU usage and log flooding. Fixed reset jobs not being able to pull images from private registries.

title: Redis Data Integration release notes 1.12.3 (July 2025)

summary: RDI Collector API now properly supports MariaDB. Extend adding custom labels and annotations to all resources in the RDI Helm chart.

title: Redis Data Integration release notes 1.12.2 (July 2025)

summary: Update documentation on pipelines endpoint. Fix deploy pipeline validation issues. /tables API can now group tables by schema. Fix mongo tables endpoint. Fix Oracle /metadata and /columns filters being case sensitive.

title: Redis Data Integration release notes 1.12.1 (June 2025)

summary: Enhanced CLI command for HA topology information. Automatic DLQ routing for key errors. Enhanced transformation job validation in the PATCH and POST /api/v1/pipelines endpoints. Added `xml_to_dict` transformation function.

title: Redis Data Integration release notes 1.12.0 (June 2025)

summary: Added support for MongoDB as a source, with a scaffold template and enhanced configuration validation. Implemented automatic deletion of pods affected by secret rotation.

title: Redis Data Integration release notes 1.10.0 (May 2025)

summary: Enhanced RDI operator for better pipeline orchestration, resiliency, observability and flexibility; External collector support; Labels and annotations to RDI data plane pods; Custom Debezium image; Calculated TTL for target database.

title: Redis Data Integration release notes 1.8.0 (May 2025)

summary: Installation on Kubernetes with a Helm chart. Improvements for installation on VMs.

title: Redis Data Integration release notes 1.6.7 (May 2025)

title: Redis Data Integration release notes 1.6.6 (April 2025)

title: Redis Data Integration release notes 1.6.5 (April 2025)

title: Redis Data Integration release notes 1.6.4 (April 2025)

title: Redis Data Integration release notes 1.6.3 (March 2025)

title: Redis Data Integration release notes 1.6.2 (March 2025)

title: Redis Data Integration release notes 1.6.1 (March 2025)

title: Redis Data Integration release notes 1.6.0 (February 2025)

title: Redis Data Integration release notes 1.4.4 (January 2025)

title: Redis Data Integration release notes 1.4.3 (December 2024)

title: Redis Data Integration release notes 1.4.2 (November 2024)

title: Redis Data Integration release notes 1.4.1 (November 2024)

title: Redis Data Integration release notes 1.4.0 (October 2024)

summary: API server with a set of APIs to support Redis Insight with creating, testing, deploying & monitoring RDI pipelines. RDI Installer verifies all components are running at the end of installation.

title: Redis Data Integration release notes 1.2.8 (August 2024)

title: Redis Data Integration release notes 1.2 (June 2024)

summary: Changes to the processing mode. Simple installation. Silent installation. Pipeline orchestration. Logging. Monitoring. High availability mechanism.

title: Redis Data Integration release notes 1.0 (June 2024)

title: Content

text: Here's what changed recently in Redis Data Integration (RDI):


title: Overview

text:
This is the first General Availability version of Redis Data Integration (RDI).

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a
[_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient
and fast data structures that match your app's requirements. You specify the
transformations using a configuration system, so no coding is required.

title: Headlines

text:
- Changes to the processing mode: The preview versions of RDI processed data inside the Redis Enterprise database using the shard CPU. The GA version moves the processing of data outside the cluster. RDI is now deployed on VMs or on Kubernetes (K8s).
- Simple installation: RDI ships with all of its dependencies. A simple interactive installer provides a streamlined process that takes a few minutes.
- Silent installation: RDI can be installed by software using a script and an input file.
- Pipeline orchestration: The preview versions of RDI required you to manually install and configure the Debezium server. In this version, we add support for source database configuration to the pipeline configuration and orchestration of all pipeline components including the Debezium server (RDI Collector).
- Logging: All RDI component logs are now shipped to a central folder and get rotated by RDI's logging mechanism.
- Monitoring: RDI comes with two Prometheus exporters, one For Debezium Server and one for RDI's pipeline data processing.
- High availability mechanism: The preview versions of RDI used an external clustering dependency to provide active-passive deployment of the Debezium server. The GA version has a Redis-based built-in fail-over mechanism between an active VM and a passive VM. Kubernetes deployments rely on K8s probes that are included in RDI components.

title: Limitations

role: limits

text:
- RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.
- RDI write-behind (which is currently in preview) should not be used on the same data set that RDI ingest is writing to Redis. This would either cause an infinite loop or would harm the data integrity, since both ingest and write-behind are asynchronous, eventually-consistent processes.


title: Redis Data Integration release notes 1.10.0 (May 2025)

title: Overview

text:
This minor release replaces the 1.8.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
- Added support for MongoDB as a source, including MongoDB 6.0 and 7.0, for both on-premises and MongoDB Atlas deployments.
- Introduced a MongoDB scaffold template and enhanced configuration validation.
- Implemented automatic deletion of pods affected by secret rotation to ensure updated secrets are applied.

title: Detailed changes

text:
**Other Features, Improvements and Enhancements**

The `redis-di scaffold` cli command can now scaffold MongoDB configuration. It can also scaffold different flavors:
  - `--db-flavor mongodb-atlas`
  - `--db-flavor mongodb-replica-set`
  - `--db-flavor mongodb-sharded-cluster`

title: Limitations

text: RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.


title: Redis Data Integration release notes 1.12.0 (June 2025)

title: Overview

text:
This minor release replaces the 1.10.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
- Enhanced CLI status command to display RDI High Availability (HA) topology information.
- Improved error handling with automatic DLQ routing for wrong Redis key errors.
- Enhanced transformation job validation in the `PATCH` and `POST` `/api/v1/pipelines`
  endpoints with better error reporting.
- Added `xml_to_dict` transformation function for converting XML strings to JSON objects.

title: Detailed changes

text:
**CLI and Monitoring**

- The `redis-di status` command now displays High Availability (HA) information for cluster status visibility.
- Fixed `rdi_version_info` metric to export periodically instead of only at startup.
- Status API: Fixed issue - pipeline status not shifting from snapshot to streaming after restart or reset

**Validation and Error Handling**

- Static configuration and job schema validation provides clearer error messages.
- Pipeline endpoints validate source and job configurations against database metadata, preventing mismatches:
  - Error in table names or table doesn't exist.
  - More than one job per table.
- Automatic DLQ routing for wrong Redis key errors.
- Support for secret rotation. Pods are recreated when a secret rotates.
- RDI validates that its Redis database is not sharded when RDI starts.

**Database Connectivity**

- MySQL connector filters system tables for cleaner metadata.
- Collector API supports multiple database and schema entries for multi-database scenarios.
- `/api/v1/pipelines/{source_name}/metadata` endpoint supports table name filtering.
- `/api/v1/pipelines/{source_name}/tables` endpoint supports table name filtering.

**API and Infrastructure**

- Pipeline endpoints (source, targets, secret-providers) support key deletion via `PATCH` operations.
- Helm chart supports lists of ingress hosts.
- Operator handles missing SPCPS CRD gracefully.

title: Limitations


title: Redis Data Integration release notes 1.12.1 (June 2025)

title: Overview

text:
This maintenance release replaces the 1.12.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
- Update documentation on pipelines endpoint.
- Fix deploy pipeline validation issues.
- `/tables` API now groups tables by schema if supported; otherwise, by database catalog.
- Fix mongo tables endpoint.
- Fix Oracle `/metadata` and `/columns` filters being case sensitive.

title: Limitations


title: Redis Data Integration release notes 1.12.2 (July 2025)

title: Overview

text:
This maintenance release replaces the 1.12.1 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
-   RDI Collector API now properly supports MariaDB.
-   Extend adding custom labels and annotations to all resources in the RDI Helm chart. Includes the webhook certificate management jobs.

title: Limitations


title: Redis Data Integration release notes 1.12.3 (July 2025)

title: Overview

text:
This maintenance release replaces the 1.12.2 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
-   Fixed operator excessive CPU usage and log flooding caused by annotation conflict between
    PipelineRelease and Kubernetes deployment controllers.
-   Fixed reset jobs not being able to pull images from private registries due to missing
    image pull secrets.

title: Limitations


title: Redis Data Integration release notes 1.14.0 (August 2025)

title: Overview

text:
RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.14.0

text:
- Added support for Google Cloud Spanner databases:
  - Compatible with both Google SQL and PostgreSQL dialects
  - Secure data collection through Apache Flink with built-in TLS/mTLS support
- Fixing: MariaDB TLS connection not working when using TLS to the RDI database
- Introduced RDI operator metrics for better monitoring and troubleshooting of the work of the operator
- The `redis-di status` command now shows a detailed breakdown of the processor processing times:
  - Separate `Transform time` metric showing how long data transformations take
  - Distinct `Write time` metric indicating Redis write operation duration

title: Limitations


title: Redis Data Integration release notes 1.14.1 (August 2025)

title: Overview

text:
This maintenance release replaces the 1.14.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.14.1

text:
- Support for Google Cloud Workload Identity authentication when a service account is assigned to the GKE cluster
- Fixed RDI API job validation that was incorrectly failing when schemas are not explicitly specified in source configuration, even though the configuration was valid

title: Limitations


title: Redis Data Integration release notes 1.15.0 (October 2025)

title: Overview

text:
RDI's mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.15.0

text:
**Breaking change when using JSON with `json_update_strategy: merge`**

RDI now uses the native `JSON.MERGE` command instead of Lua scripts for JSON merge operations. While this provides significant performance improvements (2x faster), there is a **functional difference** in how null values are handled:

- **Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)
- **New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))

**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation.

**Configuration**: You can control this behavior using the `use_native_json_merge` property in the processors section of your configuration. Set it to `false` to revert to the previous Lua-based merge behavior if needed.

- **Native JSON merge for improved performance**: RDI now automatically uses the native `JSON.MERGE` command from RedisJSON 2.6.0+ instead of Lua scripts for JSON merge operations, providing 2x performance improvement. This feature is enabled by default and can be controlled via the `use_native_json_merge` property in the processors section of the configuration. **Note**: If the target Redis database has RedisJSON version lower than 2.6.0, the processor will automatically revert to using the Lua-based merge implementation.
- **Support for sharded Redis databases**: RDI now supports writing to multi-sharded Redis Enterprise databases for the RDI database, resolving cross-slot violations when reading from streams.
- **Enhanced processor performance metrics**: Detailed performance metrics are now exposed through the metrics exporter and statistics endpoint, with separate tracking for transformation time and write time.
- **Resource management improvements**: Collector and processor pods now support configurable resource requests, limits, and node affinity/tolerations for better cluster resource utilization.
  - The `collector` defaults to 1 CPU and 1024Mi memory (requests), with limits of 4 CPUs and 4096Mi memory.
  - The `processor` defaults to 1 CPU and 512Mi memory (requests), with limits of 4 CPUs and 3072Mi memory.

- **Leadership status monitoring**: New metrics expose leadership status and pipeline phase information for better monitoring of HA deployments.
  - The `rdi_operator_is_leader` metric tracks the current leadership status of the operator: `1` indicates the instance is the leader, `0` indicates it is not the leader.
  - The `rdi_operator_pipeline_phase` metric tracks the current phase of the pipeline. Phase indicates the current pipeline phase, must be one of `Active`, `Inactive`, `Resetting`, `Pending`, or `Error`.
- **Improved configuration validation**: More rigid validation for `config.yaml` and `jobs.yaml` files helps catch configuration errors earlier in the deployment process.
- **Custom K3s installation options**: The installer now supports passing custom arguments to K3s installation for more flexible on-premises deployments.
  - Example: `sudo INSTALL_K3S_EXEC='--write-kubeconfig-mode=644' ./install.sh`
- **Workload Identity authentication**: Added support for Google Cloud Workload Identity authentication to Google Cloud Storage (GCS), eliminating the need for service account JSON files if using GSC-based leader election.
- Exposed new processor performance metrics showing transformation and write times separately
  - `{namespace}_processor_process_time_ms_total` - Total time spent in the processor (transform + write)
  - `{namespace}_processor_transform_time_ms_total` - Time spent transforming data
  - `{namespace}_processor_write_time_ms_total` - Time spent writing data to Redis
- Exposed all existing processor metrics that were only available through the RDI CLI status and the API statistics endpoint.
- Enhanced statistics endpoint with new metrics for transform and process time

title: Bug Fixes and Stability Improvements

text:
- **Fixed task reconciliation errors**: Resolved "The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict" errors during task reconciliation when using an RDI Redis database with strict XADD id uniqueness mode.
- **Fixed Debezium unavailable values**: Addressed issues where `__debezium_unavailable_value` was appearing in Redis data.
- **Improved operator stability**: Disabled operator webhooks by default to simplify deployments and reduce potential issues.

title: Limitations


title: Redis Data Integration release notes 1.15.1 (November 2025)

title: Overview

text:
This maintenance release replaces the 1.15.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.15.1

text:
- Fix dump support package problems
  - Fix improper namespace passing to the kubectl command
  - Mask passwords from sensitive configs
  - Dump masked pipeline information
  - Mask and dump audit logs collected from RDI DB rdi:tasks streams
- Processor performance improvements - Increase throughput during initial snapshot by ~50% during initial sync
  - Added parameter - `read_batch_timeout_ms` to the processor configuration to control the timeout for reading from the RDI streams. The default is 100ms.
  - `read_batch_size` is now functional. In previous versions, the processor would always read a batch size of 2000 from the stream, regardless of the `read_batch_size` value.
- Added parameter `sources.advanced.java_options` to the `config.yaml` file to pass additional options to the Debezium collector command line e.g. increasing heap size.

title: Limitations

text: RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.


title: Redis Data Integration release notes 1.16.0 (January 2026)

title: Overview

title: What's New in 1.16.0

text:
- **External secrets mount status tracking**: The operator tracks and surfaces the status of external secrets mounts, helping you diagnose issues with secrets providers more quickly.
- **Improved metrics exposure for VM installs**: RDI annotates ingresses during VM-based installations so that metrics are automatically exposed for monitoring systems.
- **Debezium upgrade**: Upgraded Debezium to version 3.3.1, bringing connector fixes and performance improvements from the upstream project.
- **Installer preflight disk space checks**: The installer now checks for sufficient disk space and `noexe` permissions on the `/tmp` directory before starting the installation, reducing failures due to insufficient capacity.

title: RDI API Changes

text:
- **Dry-run support in API v2**: RDI API v2 endpoints now supports dry-run mode, allowing you to validate pipeline configuration and connectivity without applying changes. They also now return validation errors instead of generic failures when the request body is missing or invalid.
- **Expanded API v2 observability and metadata**: RDI API v2 now exposes additional information and metrics:
  - New `/info` endpoint that returns the RDI version. This endpoint does not require authentication and can be used to check RDI availability.
  - New `/metric-collections` endpoints that describe available metric collections.
  - Pipeline `/api/v2/pipelines/{name}` and status `/api/v2/pipelines/{name}/status` responses now include component-level information.
  - Metric collection selection is now based on component type instead of name.
- **API correctness and compatibility improvements**: Ensured backwards compatibility between API v1 and v2 and fixed issues where API v2 source/target dry-run endpoints could return HTTP 500 for certain invalid requests.
- **Environment variable handling for monitoring statistics**: Fixed an issue where `/api/v1/monitoring/statistics` could return HTTP 500 when the target database port was specified as an environment variable.

title: Spanner Integration with Flink Collector

text:
- **Enhanced Spanner disaster recovery with Redis Active-Active**: When using Cloud Spanner as a CDC source with a Redis Active-Active database, the Flink collector now handles long-running workloads and failover scenarios more reliably, improving the overall ability to handle disaster recovery scenarios.
- **Spanner client resource management**: Spanner clients are now reused per database instead of being created repeatedly, eliminating a memory leak and reducing connection churn on Spanner.
- **Flink collector memory tuning**: Increased the default process and heap size for the Flink collector and tuned memory fractions for better performance and stability.
- **Resilient Redis operations by the Flink collector**: The Flink collector now uses configurable connection and socket timeouts, as well as retries with exponential backoff for Redis operations, for improved resilience. It also supports waiting for a configurable number of replicas when writing for improved durability, and handles Redis out-of-memory errors as backpressure instead of failures.

title: Bug Fixes and Stability Improvements

text:
- **Improved support package generation**: Fixed several issues in the support package dump process so that diagnostic bundles are generated reliably, including rotated logs. Added a new `--log-days` option to specify the number of days of logs to collect.
- **Fixed log level configuration**: The `redis-di configure` command now correctly applies the `rdi-log-level` setting.
- **Enhanced processor error handling**: The processor now fails with a clear error when it cannot connect to the target database after a timeout and writes such errors to the pod termination log to expose them.
- **Improved pipeline error detection**: Pipelines now transition to an `Error` status when deployments are not available after 2 minutes, exposing failures sooner.
- **Fixed empty password handling**: Resolved issues with RDI commands when using empty passwords.
- **Operator namespace isolation**: The RDI operator now only reconciles resources within the RDI namespace, improving performance and isolation. It also no longer requires cluster-level permissions for any resources, reducing security concerns for enterprise Kubernetes deployments.

title: Limitations


title: Redis Data Integration release notes 1.16.1 (February 2026)

title: Overview

text:
This maintenance release replaces the 1.16.0 release.

RDI's mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.16.1

text:
- **Improved CA certificate chain handling from external secrets**: When importing CA certificates from external secrets mounts, RDI now correctly loads the full certificate chain instead of only the first certificate. This ensures proper trust configuration for TLS connections that rely on intermediate CAs.
- **More flexible API v2 pipeline patching**: Fixed API v2 pipeline PATCH behavior so that you can remove tables and table properties by sending `null` values. This aligns the API with the documented semantics for partial updates and makes it easier to clean up pipeline configurations.
- **Security hardening for collector API dependencies**: Updated the collector API to newer, supported versions of key dependencies to address multiple CVEs and improve overall security posture, including:
  - Spring Boot (to 3.3.11), which also brings updated Spring Framework and Spring Security components.
  - Netty (to 4.1.125.Final).
  - Kafka clients (to 3.9.1).
  - MySQL Connector/J (to 8.2.0).
  - Logback (to 1.5.18) and Jetty (to 12.0.19) via the Spring Boot upgrade.

title: Limitations


title: Redis Data Integration release notes 1.16.2 (March 2026)

title: Overview

text:
This maintenance release replaces the 1.16.1 release.

RDI's mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: What's New in 1.16.2

text: - **New configuration option to disable AOF check:** A new prerequisiteChecks section is introduced in the operator section of the Helm values file. You can use it to disable the AOF prerequisite check when running RDI against a Redis RDI database that does not have AOF enabled. Please note that disabling this check without understanding the implications may lead to data loss in certain failure scenarios, so it should be used with caution.

title: Limitations


title: Overview

text:
> This minor release replaces the 1.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a
[_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient
and fast data structures that match your app's requirements. You specify the
transformations using a configuration system, so no coding is required.

title: Headlines

text:
- API server with a set of APIs to support Redis Insight with creating, testing, deploying & monitoring RDI pipelines.
- RDI Installer verifies all components are running at the end of installation.

title: Fixed Bugs

text:
- Support for source database TLS & mTLS was fixed. Certificate and file names used with the redis-di set-secret command can have any file names
- Mismatch between Reloader version provided and the one required.

title: Limitations

text:
- RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.
- RDI write-behind (which is currently in preview) should not be used on the same data set that RDI ingest is writing to Redis. This would either cause an infinite loop or would harm the data integrity, since both ingest and write-behind are asynchronous, eventually-consistent processes.


title: Overview

text:
# Redis Data Integration 1.2.8 GA

> This maintenance release replaces the 1.2 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

title: Fixed Bugs

text:
- Fixed issue with [CoreDNS](https://coredns.io/) configuration. Previously, it could only resolve from the host DNS.
- Fixed incorrect validation schema for jobs in [Redis Insight](https://redis.io/docs/latest/develop/connect/insight/).
- The pipeline advanced settings were missing when downloading the pipeline from RDI but this is now fixed.

title: Limitations


title: Redis Data Integration release notes 1.4.0 (October 2024)

title: Overview

text:
> This is a GA version of Redis Data Integration (RDI) that improves the installation of RDI.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
-   Installation on [Kubernetes]() using a [Helm chart](https://helm.sh/docs/). You can install on [OpenShift](https://docs.openshift.com/) or other flavours of K8s using Helm.

- Improvements for installation on VMs:
	- Installer checks if the OS firewall is enabled on Ubuntu and RHEL.
	- Installer verifies DNS resolution from RDI components.
	- Installer provides log lines from components that failed during RDI deployment if a problem occurs.
	- Improved verification of RDI installation.
	- Installer verifies if the RDI database is in use by another instance of RDI.
	- Installer checks and warns if any [`iptables`](https://www.netfilter.org/projects/iptables/index.html) rules are set.
	- Improved message when RDI tries to connect to its Redis database with invalid TLS keys.

title: Limitations


title: Redis Data Integration release notes 1.4.1 (November 2024)

title: Overview

text:
> This maintenance release replaces the 1.4.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

title: Issues fixed

text:
- **RDSC-2806**: Remove incorrectly created deployment.
- **RDSC-2792**: Disable `kubectl run` checks.
- **RDSC-2782**: Fix `coredns` issue.

title: Limitations

text: RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.


title: Redis Data Integration release notes 1.4.2 (November 2024)

title: Overview

text:
> This maintenance release replaces the 1.4.1 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

title: Issues fixed

text:
- **RDSC-2802**: Reintroduce checks for DNS resolution and network connectivity from inside a K8s pod.
- **RDSC-2804**: RDI installation previously failed if the user specified a non-default HTTPS port.
  The user can now specify the port correctly.

title: Limitations


title: Redis Data Integration release notes 1.4.3 (December 2024)

title: Overview

text:
> This maintenance release replaces the 1.4.2 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

title: Issues fixed

text:
- **RDSC-2963**: Helm chart `rdiSysSecret` does not create an empty secret if you are not using a password.
- **RDSC-2729**: Use Debezium Server 2.7.3 and remove Prometheus.
- **RDSC-2333**: Ensure the installer creates the context file correctly.
- **RDSC-2806**: Remove incorrectly created deployment.
- **RDSC-2729**: Fix `processors` `null` instead of `{}`.
- **RDSC-2905**: Fix DNS check with multiple IP addresses.
- **RDSC-2845**: RDI Helm chart release is set with `tag: 0.0.0`, but this should be the current release.

title: Limitations


title: Redis Data Integration release notes 1.4.4 (January 2025)

title: Overview

text:
> This maintenance release replaces the 1.4.3 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
-   Installation on [Kubernetes]() using a [Helm chart](https://helm.sh/docs/). You can install on [OpenShift](https://docs.openshift.com/) or other flavours of K8s using Helm.

- Improvements for installation on VMs:
  - Adding an `upgrade.sh` script
  - Uninstall script removes RDI CLI
  - Adding an ingress to the `collector-source` metrics exporter for VM installs
  - Fix RDI version issues during upgrade
  - Ensure `KUBECONFIG` is set during upgrade
  - Upgrade [`datayoga`](https://github.com/datayoga-io/datayoga) to 1.127.0 - add_field block was rejected for one or more records but the entire batch was marked as rejected. making troubleshooting difficult
  - Fix installer not setting `RDI_REDIS_SSL` properly if SSL is enabled

title: Issues fixed

text:
- **RDSC-3103**: If a record's transformation is rejected, the entire Processor batch ends up in DLQ
- **RDSC-3130**: Failed to install 1.4.3, but successful install with 1.2.8
- **RDSC-3141**: There is no way to reach metrics when running RDI on VM
- **RDSC-3142**: `redis-di upgrade` fails with error message - CRITICAL - Error while attempting to upgrade RDI: Could not get the current version of the RDI instance
- **RDSC-3143**: RDI on VM upgrade error message - Get "http://localhost:8080/openapi/v2?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
- **RDSC-3156**: Add script to execute `redis-di upgrade` correctly

title: Limitations


title: Redis Data Integration release notes 1.6.0 (February 2025)

title: Overview

text:
> This maintenance release replaces the 1.4.4 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
- RDI now requires the RDI database to have the following properties set, otherwise
  RDI will not start:
  - `maxmemory_policy`: `noeviction`,
  - `aof_enabled`: `1`

- Allow RDI to run in any K8s namespace

- Fix metadata API to support Oracle and SQL Server

- Added [denormalisation lookup block]()

- Many bug fixes

title: Limitations


title: Redis Data Integration release notes 1.6.1 (March 2025)

title: Overview

text:
> This release replaces the 1.6.0 release.

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

title: Headlines

text:
- Fixed missing metrics and incorrect status for SQL Server.
- Improved Redis Insight autocomplete using schema description property.
- Restricted Redis connections to targets and non-Redis connections to sources.
- Enhanced scaffolded `config.yaml` with examples for each database type.
- VM Installer: Captured output of all sub-processes.
- Improved error message for critical errors when sending commands to Redis cluster fails.
- Fixed issue where pipeline status did not return to streaming after stopping and starting.
- Added RDI API logging, validation examples, and bug fixes.
- Disabled strict parsing of collector application properties to allow overriding.
- Fixed HTTP 500 error in `GET /pipelines/sources/{source}/columns` when passing tables parameter.
- VM setup: Fixed incorrect output status and incomplete upgrades in `upgrade.sh` script.
- Ensured `collector-api` service force upgrades when `Operator` restarts.
- Corrected typos in CLI option help messages.
- Generated and published [API reference page]().
- Resolved error in `collector-api` when Source secret contains special characters.
- Fixed RDI 1.6.0 installation failure for HA due to `rdi-operator` deployment timeout.
- Added support for downloading Debezium image from a private registry.
- VM Installer now supports Ubuntu 22.04 and Ubuntu 24.04.
- Resolved performance degradation in 1.6.0 that caused the initial sync to be 3-5 times slower.

title: Limitations