Redis Data Integration configuration file
Redis Data Integration configuration file reference
Configuration file for Redis Data Integration (RDI) source collectors and target connections
Properties
Name | Type | Description | Required |
---|---|---|---|
sources (Source collectors) |
object |
Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector |
|
processors (Data processing configuration) |
object , null |
Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning |
|
targets (Target connections) |
object |
Configuration for target Redis databases where processed data will be written |
|
secret-providers (Secret providers) |
object |
sources: Source collectors
Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
Properties (Pattern)
Name | Type | Description | Required |
---|---|---|---|
.* |
processors: Data processing configuration
Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
Properties
Name | Type | Description | Required |
---|---|---|---|
on_failed_retry_interval (Retry interval on failure) |
integer , string |
Number of seconds to wait before retrying a failed operation Default: 5 Pattern: ^\${.*}$ Minimum: 1 |
|
read_batch_size | integer , string |
Maximum number of records to process in a single batch Default: 2000 Pattern: ^\${.*}$ Minimum: 1 |
|
debezium_lob_encoded_placeholder (Debezium LOB placeholder) |
string |
Placeholder value for LOB fields in Debezium Default: "__debezium_unavailable_value" |
|
dedup (Enable deduplication) |
boolean |
Enable the deduplication mechanism to handle duplicate records Default: false |
|
dedup_max_size (Deduplication set size) |
integer |
Maximum number of entries to store in the deduplication set Default: 1024 Minimum: 1 |
|
dedup_strategy (Deduplication strategy) |
string |
(DEPRECATED) Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration. Default: "ignore" Enum: "reject" , "ignore" |
|
duration (Batch duration limit) |
integer , string |
Maximum time in milliseconds to wait for a batch to fill before processing Default: 100 Pattern: ^\${.*}$ Minimum: 1 |
|
write_batch_size | integer , string |
Maximum number of records to write to target Redis database in a single batch Default: 200 Pattern: ^\${.*}$ Minimum: 1 |
|
error_handling (Error handling strategy) |
string |
Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue Default: "dlq" Pattern: ^\${.*}$|ignore|dlq |
|
dlq_max_messages (DLQ message limit) |
integer , string |
Maximum number of messages to store in dead letter queue per stream Default: 1000 Pattern: ^\${.*}$ Minimum: 1 |
|
target_data_type (Target Redis data type) |
string |
Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module) Default: "hash" Pattern: ^\${.*}$|hash|json |
|
json_update_strategy | string |
(DEPRECATED) Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy. Default: "replace" Pattern: ^\${.*}$|replace|merge |
|
initial_sync_processes | integer , string |
Number of parallel processes for performing initial data synchronization Default: 4 Pattern: ^\${.*}$ Minimum: 1 Maximum: 32 |
|
idle_sleep_time_ms (Idle sleep interval) |
integer , string |
Time in milliseconds to sleep between processing batches when idle Default: 200 Pattern: ^\${.*}$ Minimum: 1 Maximum: 999999 |
|
idle_streams_check_interval_ms (Idle streams check interval) |
integer , string |
Time in milliseconds between checking for new streams when processor is idle Default: 1000 Pattern: ^\${.*}$ Minimum: 1 Maximum: 999999 |
|
busy_streams_check_interval_ms (Busy streams check interval) |
integer , string |
Time in milliseconds between checking for new streams when processor is busy Default: 5000 Pattern: ^\${.*}$ Minimum: 1 Maximum: 999999 |
|
wait_enabled (Enable replica wait) |
boolean |
Enable verification that data has been written to replica shards Default: false |
|
wait_timeout (Replica wait timeout) |
integer , string |
Maximum time in milliseconds to wait for replica write verification Default: 1000 Pattern: ^\${.*}$ Minimum: 1 |
|
retry_on_replica_failure | boolean |
Continue retrying writes until successful replication to replica shards is confirmed Default: true |
Additional Properties: not allowed
targets: Target connections
Configuration for target Redis databases where processed data will be written
Properties (Pattern)
Name | Type | Description | Required |
---|---|---|---|
.* |
secret-providers: Secret providers
Properties (Pattern)
Name | Type | Description | Required |
---|---|---|---|
.* (Secret provider entry) |
object |
yes |
secret-providers..*: Secret provider entry
Properties
Name | Type | Description | Required |
---|---|---|---|
type (Provider type) |
string |
Enum: "aws" , "vault" |
yes |
parameters (Provider parameters) |
object |
yes |
Additional Properties: not allowed
Example
parameters:
objects:
- {}
secret-providers..*.parameters: Provider parameters
Properties
Name | Type | Description | Required |
---|---|---|---|
objects (Secrets objects array) |
object[] |
yes |
Example
objects:
- {}
secret-providers..*.parameters.objects[]: Secrets objects array
Items: Secret object
No properties.
Example
- {}