Redis Data Integration release notes 1.8.0 (May 2025)
Enhanced RDI operator for better pipeline orchestration, resiliency, observability and flexibility; External collector support; Labels and annotations to RDI data plane pods; Custom Debezium image; Calculated TTL for target database.
RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:
- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.
RDI keeps the Redis cache up to date with changes in the primary database, using a Change Data Capture (CDC) mechanism. It also lets you transform the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.
Headlines
- Enhanced RDI operator for better pipeline orchestration, resiliency, observability, and flexibility. It will also enable many new features in the near future.
- You can now use an external collector that is not managed by RDI but writes into RDI streams (Debezium compatible).
- You can now add labels and annotations to RDI data plane pods, for example to control service mesh features.
- RDI now uses a custom image of Debezium (based on
3.0.8.Final
) to address known vulnerabilities. - Added support for calculated TTL for target database keys via
expire
expressions.
Detailed changes
Helm chart changes
- All collector and processor values that were previously under
collector
,collectorSourceMetricsExporter
, andprocessor
have been moved tooperator.dataPlane.collector
andoperator.dataPlane.processor
. global.collectorApiEnabled
has been moved tooperator.dataPlane.collectorApi.enabled
, and is now a boolean value (true
orfalse
), not"0"
or"1"
.api.authEnabled
is also now a boolean value, not"0"
or"1"
.- The following values have been deprecated:
rdiMetricsExporter.service.protocol
,rdiMetricsExporter.service.port
,rdiMetricsExporter.serviceMonitor.path
,api.service.name
- You can now add custom labels and annotations to all RDI components.
- You can now disable the creation of the RDI system secrets.
Operator Improvements
The RDI operator has been significantly enhanced in the following areas:
- Resilience: The operator now always maintains the desired pipeline state. Manual changes or random disruptions are reverted automatically.
- Automatic recovery: When a configuration issue is resolved, the entire pipeline starts automatically, eliminating the need for manual redeployment.
- Consistency: A pipeline that has been stopped with
stop
will remain stopped afterdeploy
orreset
, until explicitly started again. - Enhanced configuration: You can now configure data plane components in ways that were previously not supported, such as adding labels and annotations.
- External collector support: No collector resources are created for sources of type
external
. - Enhanced troubleshooting: You can now gain extra insight into the pipeline state by examining the
Pipeline
andPipelineRelease
custom K8s resources.
Other Features, Improvements and Enhancements
- Added
expire
expression for target output in transformation jobs. - Addressed security vulnerabilities: TLS certificate hostname verification is now ON by default.
- Improved Helm default values while preserving
values.yaml
formatting. - Enhanced Helm values and templates for better configuration.
- Added a script to create or update secrets when using Helm (
rdi-secret.sh
in the Helm zip file). - Improved validation schema and ensured backward compatibility.
- Fixed compatibility issues with newer versions of
requests
andurllib3
. - Improved error messages for JSON schema validation.
- Improved PostgreSQL documentation for mTLS.
- Added timestamps to the
status
command. - Fixed issues with
primary_key
andunique_constraint
attributes in Oracle metadata. - Added
capture.mode
to MongoDB scaffolding. - Improved Helm TLS setup for RDI database connections.
- Enhanced error handling and validation for transformation jobs.
- Improved documentation for supported platforms and configurations.
Fixes
- Fixed HTTP 500 error when querying columns with tables parameter.
- Improved Helm TLS setup for RDI database connections.
- Fixed keystore overwrite when using mTLS on both source and RDI DBs in the collector.
Limitations
RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.