title: Write-behind configuration guide

summary: Configure write-behind to your database

title: Overview

text: This guide shows you how to configure write-behind target connections.

title: Overview

text:
Write-behind target connections are the connections established between a Write-behind instance and a target database in a
[write-behind scenario]().
Write-behind is used to replicate changes captured in a Write-behind-enabled Redis Enterprise database to a target database.
The connections must be configured in the `config.yaml` before deploying any jobs and must follow one of the formats shown below. Multiple connections can be specified in the `connections` section.

title: For relational datastores

text: [code example]

title: For non-relational datastores

text: [code example]

title: Microsoft SQL Server

text: Microsoft SQL Server supports different authentication mechanisms (SQL Server Authentication and Integrated Windows Authentication) and protocols (NTLM and Kerberos). Write-behind can use all of them. However, systems that use Kerberos may require some additional configuration.

title: Account permissions

text: To enable Write-behind to work with a SQL Server database, check that the account you specify was assigned at least the `db_datawriter` role.

title: SQL Server authentication

text:
To use SQL Server authentication mode, create a user with login credentials and then assign the necessary permissions for the target database to that user.

[code example]

title: Windows authentication

text:
To use Windows authentication mode, you need to create a Windows or Active Directory account that has the necessary permissions to access the target database, and is able to log into SQL Server. The Linux machine hosting Write-behind can be configured to support the NTLM authentication protocol.

For NTLM:

[code example]

> Note: User must be specified with the domain name for Windows Authentication to work correctly.

After you configure the Write-behind connection and deploy the write-behind job, run the following SQL query to have the operator check if Write-behind is using the expected authentication mechanism and protocol. Note: this operation may require the `sysadmin` role.

[code example]

The results indicate which `auth_scheme` is used by each session and may take values `SQL`, `NTLM`, and `Kerberos`.

code:
connections:
    my-sql-datastore:
        type: <db-type>     # mysql | oracle | postgresql | sqlserver
        host: <db-host>     # IP address or FQDN of a database host and instance
        port: <db-port>     # database port
        database: <db-name> # name of the database
        user: <db-user>         # database user
        password: <db-password> # database password
        # connect_args:     # optional connection parameters passed to the driver - these are driver specific
        # query_args:       # optional parameters for SQL query execution - typically not required for Write-behind operation

section_id: for-relational-datastores

code:
connections:
    my-nosql-datastore:
        type: <db-type>     # cassandra
        hosts: <db-hosts>   # array of IP addresses or host names of a datastore nodes
        port: <db-port>     # database port
        database: <db-name> # name of the database
        user: <db-user>         # database user
        password: <db-password> # database password

section_id: for-non-relational-datastores

code:
connections:
    mssql2019-sqlauth:
        type: sqlserver
        host: ip-10-0-0-5.internal
        port: 1433
        database: rdi_wb_database
        user: rdi_user
        password: secret

section_id: sql-server-authentication

code:
connections:
    mssql2019-ntlm:
        type: sqlserver
        host: ip-10-0-0-5.internal
        port: 1433
        database: rdi_wb_database
        user: MYDOMAIN\rdi_service_account  # company-domain\service-account
        password: secret                    # NTLM requires to provide a password

section_id: windows-authentication

code: SELECT session_id, auth_scheme FROM sys.dm_exec_connections;


title: Data transformation pipeline

summary: Learn how to transform data to Redis types

page_type: content

title: Overview

role: overview

text:
Write-behind's data transformation capabilities allow users to transform their data beyond the default translation of source types to Redis types. The transformation involves no coding. Instead, it is described in a set of human readable YAML files, one per source table.

The ingested format and types are different from one source to another. Currently, the only supported source is [Debezium](https://debezium.io/). The first transformation from Debezium types to native JSON with Redis types is done automatically without any need for user instructions. Then, this JSON is passed on to the user defined transformation pipeline.

Each job describes the transformation logic to perform on data from a single source. The source is typically a database table or collection and is specified as the full name of this table/collection. The job may include filtering logic to skip data that matches a condition. Other logical steps in the job will transform the data into the desired output that will be stored in Redis as hashes or JSON.

title: Default job

role: content

text:
In situations where there is a need to perform a transformation on all ingested records without creating a specific job for specific tables, the default job is used. The transformation associated with this job will be applied to all tables that lack their own explicitly defined jobs. The default job must have a table name of "*", and only one instance of this type of job is permitted.

For example, the default job can streamline tasks such as adding a prefix or postfix to Redis keys, or adding fields to new hashes and JSONs without customizing each source table.

Currently, the default job is supported for ingest pipelines only.

title: Example

role: example

text:
This example demonstrates the process of adding an `app_code` field with a value of `foo` using the [add_field]() block to all tables that lack explicitly defined jobs. Additionally, it appends an `aws` prefix and a `gcp` postfix to every generated hash key.

default.yaml
[code example]

title: Jobs

text:
Each job is defined in a separate YAML file. All of these files will be uploaded to Write-behind using the `deploy` command.
For more information, see [deploy configuration](#deploy-configuration)). If you are using the
[scaffold]() command,
place the job files in the `jobs` folder.

title: Job YAML structure

text:
#### Fields

- `source`:

  This section describes the table that the job operates on:

  - `server_name`: logical server name (optional). Corresponds to the `debezium.source.topic.prefix` property specified in Debezium Server's `application.properties` config file
  - `db`: database name (optional)
  - `schema`: database schema (optional)
  - `table`: database table name
  - `row_format`: format of the data to be transformed: `data_only` (default) - only payload, full - complete change record

> Note: Any reference to the properties `server_name`, `db`, `schema`, and `table` will be treated by default as case insensitive. This can be changed by setting `case_insensitive` to `false`.

> Cassandra only: In Cassandra, a `keyspace` is roughly the equivalent to a `schema` in other databases. Write-behind uses the `schema` property declared in a job file to match the `keyspace` attribute of the incoming change record.

> MongoDB only: In MongoDB, a `replica set` is a cluster of shards with data and can be regarded as roughly equivalent to a `schema` in a relational database. A MongoDB `collection` is similar to a `table` in other databases. Write-behind uses the `schema` and `table` properties declared in a job file to match the `replica set` and `collection` attributes of the incoming change record, respectively.

- `transform`:

  This section includes a series of blocks that define how the data will be transformed.
  For more information, see
  [supported blocks]()
  and [JMESPath custom functions]().

- `output`:

  This section defines the output targets for processed data:

  - Cassandra:
    - `uses`: `cassandra.write`: write into a Cassandra data store
    - `with`:
      - `connection`: connection name
      - `keyspace`: keyspace
      - `table`: target table
      - `keys`: array of key columns
      - `mapping`: array of mapping columns
      - `opcode_field`: the name of the field in the payload that holds the operation (c - create, d - delete, u - update) for this record in the database
  - Redis:
    - `uses`: `redis.write`: write to a Redis data structure. Multiple blocks of this type are allowed in the same job
    - `with`:
      - `connection`: connection name as defined in `config.yaml` (by default, the connection named 'target' is used)
      - `data_type`: target data structure when writing data to Redis (hash, json, set and stream are supported values)
      - `key`: this allows you to override the key of the record by applying custom logic:
        - `expression`: expression to execute
        - `language`: expression language, JMESPath or SQL
      - `expire`: positive integer value indicating a number of seconds for the key to expire. If not set, the key will never expire
  - SQL:
    - `uses`: `relational.write`: write into a SQL-compatible data store
    - `with`:
      - `connection`: connection name
      - `schema`: schema
      - `table`: target table name
      - `keys`: array of key columns
      - `mapping`: array of mapping columns
      - `opcode_field`: the name of the field in the payload that holds the operation (c - create, d - delete, u - update) for this record in the database

#### Notes

- `source` is required.
- Either `transform`, `key`, or both should be specified.

#### Using key in transformations

To access the Redis key (for example in a write-behind job) you will need to take the following steps:

- Set `row_format: full` to allow access to the key that is part of the full data entry.
- Use the expression `key.key` to get the Redis key as a string.

#### Before and after values

Update events typically report `before` and `after` sections, providing access to the data state before and after the update.
To access the "before" values explicitly, you will need to:

- Set `row_format: full` to allow access to the key that is part of the full data entry.
- Use the `before.<FIELD_NAME>` pattern.

title: Example

text:
This example shows how to rename the `fname` field to `first_name` in the table `emp` using the `rename_field` block. It also demonstrates how to set the key of this record instead of relying on the default logic.

redislabs.dbo.emp.yaml

[code example]

title: Deploy configuration

text:
In order to deploy your jobs to the remote Write-behind database, run:

[code example]

title: Deploy configuration on Kubernetes

text:
If the Write-behind CLI is deployed as a pod in a Kubernetes cluster, perform these steps to deploy your jobs:

- Create a ConfigMap from the YAML files in your `jobs` folder:

  [code example]

- Deploy your jobs:

  [code example]

> Note: A delay occurs between creating/modifying the ConfigMap and its availability in the `redis-di-cli` pod. Wait around 30 seconds before running the `redis-di deploy` command.

You have two options to update the ConfigMap:

- For smaller changes, you can edit the ConfigMap directly with this command:

  [code example]

- For bigger changes such as adding another job file, edit the files in your local `jobs` folder and then run this command:

  [code example]

> Note: You need to run `kubectl exec -it pod/redis-di-cli -- redis-di deploy` after updating the ConfigMap with either option.

code:
source:
  table: "*"
  row_format: full
transform:
  - uses: add_field
    with:
      fields:
        - field: after.app_code
          expression: "`foo`"
          language: jmespath
output:
  - uses: redis.write
    with:
      data_type: hash
      key:
        expression: concat(['aws', '#', table, '#', keys(key)[0], '#', values(key)[0], '#gcp'])
        language: jmespath

section_id: example

code:
source:
  server_name: redislabs
  schema: dbo
  table: emp
transform:
  - uses: rename_field
    with:
      from_field: fname
      to_field: first_name
output:
  - uses: redis.write
    with:
      connection: target
      key:
        expression: concat(['emp:fname:',fname,':lname:',lname])
        language: jmespath

code: redis-di deploy

section_id: deploy-configuration

section_id: deploy-configuration-on-kubernetes

code: kubectl edit configmap redis-di-jobs

code: kubectl create configmap redis-di-jobs --from-file=jobs/ --dry-run=client -o yaml | kubectl apply -f -


title: Data transformation

title: Data transformation pipeline

summary: See examples of write-behind transform job configurations

title: Write-behind transformation examples

title: Overview

text:
The key functionality that Write-behind performs is mapping the data coming from [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) (representing a Source Database row data or row state change) into a Redis key with a value of [Hash]() or [JSON]().
There are two types of data transformations in Write-behind:

1. By default, each source row is converted into one hash or one JSON key in Redis.
  This conversion uses the Debezium schema-based conversion. The incoming data includes the schema and Write-behind uses a set of handlers to automatically convert each source column to a Redis Hash field or JSON type based on the Debezium type in the schema. See
  [data type conversion]()
  for a full reference on these conversions.

1. If the user wants to add or modify this default mapping, Write-behind provides declarative data transformations. These transformations are represented in YAML files. Each file contains a job, which is a set of transformations per source table. See
[declarative transformations]() for more information.

title: More info


title: Write-behind foreach example

title: Content

text:
The `foreach` section is used to explode a list of objects or arrays to rows in a selected target.
The `foreach` expression is structured as `<field_name>:<JMESPath expression>`.
The following example uses the `add_field` transformation to prepare the input JSON to the desired structure. Then, it applies `foreach` to write each `order` object as a relational database record using `keys` and `mapping`.
In this example, the `JMESPath` function `to_string` is used to flatten an array of objects, `specs`, to a string.

[code example]

code:
source:
  redis:
    key_pattern: orderdetail:*
    trigger: write-behind
    exclude_commands: ["json.del"]
transform:
  - uses: add_field
    with:
      fields:
        - field: my_orders
          language: jmespath
          expression: |
            orders[*].{
              code: code
              periodStartTime: periodStartTime
              periodEndTime: periodEndTime
              specs: to_string(specs)
            }
output:
  - uses: relational.write
    with:
      connection: mssql
      schema: dbo
      table: OrderMaster
      keys:
        - Code: orderDetail.code
      mapping:
        - DeliveryDate: orderDetail.deliveryDate
        - ProductCode: orderDetail.productCode
        - CountryCode: orderDetail.countryCode
  - uses: relational.write
    with:
      connection: mssql
      schema: dbo
      table: Order
      foreach: "order: my_orders[]"
      keys:
        - Code: order.code
      mapping:
        - OrderDetailCode: orderDetail.code
        - PeriodStartTime: order.periodStartTime
        - PeriodEndTime: order.periodEndTime
        - Specs: order.specs

section_id: content


title: Write-behind transformation examples

title: Write-behind foreach example

title: Write-behind to Redis Enterprise target example

page_type: index


title: Content

text:
The `redis.write` block can be used in the `output` section of the write-behind job in order to enable writing data to the Redis Enterprise target database. Multiple blocks can be used at the same time to write data to different data types. The following example captures data modified in the `address:*` keyspace, then creates a new JSON-like string field named `json_addr`,  and, finally, writes the results to multiple keys in target Redis Enteprise database:

[code example]

Run the following command in the source Redis Enterprise database to test the job:

[code example]

The result is five keys will be created in the target Redis Enterprise database (hash, JSON, set, sorted set, and stream):

[code example]

code:
source:
  redis:
    trigger: write-behind
    key_pattern: address:*
transform:
  - uses: add_field
    with:
      fields:
        - field: "json_addr"
          expression: concat(['{"city":', city, ', "zip":', zip, '}'])
          language: jmespath
output:
  - uses: redis.write
    with:
      data_type: hash
      connection: target
      key:
        expression: concat(['addr:org_id:', org_id, ':hash'])
        language: jmespath
  - uses: redis.write
    with:
      data_type: json
      key:
        expression: concat(['addr:org_id:', org_id, ':json'])
        language: jmespath
      on_update: merge
  - uses: redis.write
    with:
      data_type: set
      key:
        expression: concat(['addresses:', country])
        language: jmespath
      args:
        member: json_addr
  - uses: redis.write
    with:
      data_type: sorted_set
      key:
        expression: concat(['addrs_withscores:', country])
        language: jmespath
      args:
        score: zip
        member: json_addr
  - uses: redis.write
    with:
      data_type: stream
      key:
        expression: "`addresses:events`"
        language: jmespath
      mapping:
        - org_id: message_id
        - zip: zip_code
        - country

code: 127.0.0.1:12005> hset address:1 city Austin zip 78901 org_id 1 country USA

code:
127.0.0.1:12000> keys *
1) "addr:org_id:1:hash"
2) "addr:org_id:1:json"
3) "addresses:USA"
4) "addrs_withscores:USA"
5) "addresses:events"


title: Installation

summary: Learn how to install Write-behind

summary: Install Write-behind CLI

title: Install Write-behind CLI

summary: Install and set up RedisGears for a Write-behind deployment

title: Install RedisGears for Redis Data Integration


title: Install Write-behind CLI

summary: Install Write-behind CLI

title: Overview

text:
The following installation instructions install the Write-behind CLI on a local workstation.

Write-behind installation is done via the Write-behind CLI. The CLI should have network access to the Redis Enterprise cluster API (port 9443 by default).

title: Download Write-behind CLI

text:
#### Ubuntu 20.04

[code example]

#### Ubuntu 18.04

[code example]

#### RHEL 8

[code example]

#### RHEL 7

[code example]

title: Install Write-behind CLI

role: setup

text:
Unpack the downloaded `redis-di.tar.gz` into the `/usr/local/bin/` directory:

[code example]

> Note: Non-root users should unpack to a directory with write permission and run `redis-di` directly from it.

code: wget https://qa-onprem.s3.amazonaws.com/redis-di//redis-di-ubuntu20.04-.tar.gz -O /tmp/redis-di.tar.gz

section_id: download-write-behind-cli

code: wget https://qa-onprem.s3.amazonaws.com/redis-di//redis-di-ubuntu18.04-.tar.gz -O /tmp/redis-di.tar.gz

code: wget https://qa-onprem.s3.amazonaws.com/redis-di//redis-di-rhel8-.tar.gz -O /tmp/redis-di.tar.gz

code: wget https://qa-onprem.s3.amazonaws.com/redis-di//redis-di-rhel7-.tar.gz -O /tmp/redis-di.tar.gz

code: sudo tar xvf /tmp/redis-di.tar.gz -C /usr/local/bin/

section_id: install-write-behind-cli


title: Overview

text:
Write-behind requires that [RedisGears](https://redis.com/modules/redis-gears) module with the [Python plugin](https://docs.redis.com/latest/modules/redisgears/python/) is installed on the Redis Enterprise cluster.

The Python plugin can be installed explicitly or alongside with the [JVM plugin](https://docs.redis.com/latest/modules/redisgears/jvm/) if the latter is needed on the cluster for other purposes.

Use the [`redis-di create`]() command in Write-behind CLI to install RedisGears.

title: Download RedisGears

text: Download RedisGears based on the Linux distribution of where Redis Enterprise is installed.

title: Ubuntu 20.04

text: [code example]

title: Ubuntu 18.04

text: [code example]

title: RHEL8

text: [code example]

title: RHEL7

text: [code example]

code: curl -s --tlsv1.3 https://redismodules.s3.amazonaws.com/redisgears/redisgears.Linux-ubuntu20.04-x86_64.-withdeps.zip -o /tmp/redis-gears.zip

code: curl -s --tlsv1.3 https://redismodules.s3.amazonaws.com/redisgears/redisgears.Linux-ubuntu18.04-x86_64.-withdeps.zip -o /tmp/redis-gears.zip

code: curl -s https://redismodules.s3.amazonaws.com/redisgears/redisgears.Linux-rhel8-x86_64.-withdeps.zip -o /tmp/redis-gears.zip

code: curl -s https://redismodules.s3.amazonaws.com/redisgears/redisgears.Linux-rhel7-x86_64.-withdeps.zip -o /tmp/redis-gears.zip


title: Monitoring guide

summary: Monitor Write-behind engine and data processing jobs

title: Overview

text:
Write-behind accumulates operating statistics that you can:

- Observe and analyze to discover various types of problems.
- Use for optimization purposes.

title: Console metrics

text:
Write-behind can display its operating metrics in the console using the
[`redis-di status`]()
command. The command provides information about the current Write-behind engine status, target database configuration, and processing statistics broken down by stream. This tool is intended to be used by Operator to get the current snapshot of the system as well as monitoring ongoing data processing (when used in live mode).

title: Prometheus integration

text:
Write-behind allows collecting and exporting its metrics to [Prometheus](https://prometheus.io/) and visualizing them in [Grafana](https://grafana.com/). Operator can start the built-in exporter using the
[`redis-di monitor`]()
command. The diagram describes this flow and the components involved:

> Note: The host names and ports above are examples only and can be changed as needed.

title: Test Write-behind metrics exporter

text:
Start the Write-behind metrics exporter using the command below:

[code example]

> Note: The default port for the exporter is `9121`. If you need to change it, use the `--exporter-port` option. The default metrics collection interval is 5 seconds. If you need to change it, use the `--collect-interval` option.

Then navigate to `http://localhost:9121/` to see the exported metrics. You should be able to see the following metric:

[code example]

> Note: The actual value of the metric above can be 0 if you haven't started Write-behind engine yet (in which case, the `state` label should indicate that as well). You must have the Write-behind database created and configured before observing any metrics. If you are not seeing it or getting an error value instead, this indicates that the Write-behind database is not properly configured.

title: Configure Prometheus

role: configuration

text:
Next, configure the Prometheus scraper. Edit the `prometheus.yml` file to add the following scraper config:

[code example]

> Notes:

- Make sure the `targets` value above points to the host and port you configured to run the Write-behind metrics exporter.
- The `scrape_interval` setting in Prometheus should be the same or higher than the `collect_interval` setting for the exporter. For example, if the `collect_interval` is set to 5 seconds, the `scrape_interval` should also be set to 5 seconds or more. If the `scrape_interval` is set to less than the `collect_interval`, Prometheus will scrape the exporter before it has a chance to collect and refresh metrics, and you will see duplicated values in Prometheus.

title: Test the Prometheus scraper

text:
After the scraper config is added to the Prometheus configuration, you should now be able to navigate to `http://<HOSTNAME>:9090/graph` (replace `<HOSTNAME>` with a valid Prometheus hostname or IP address).

Explore Write-behind metrics using the [expression browser](https://prometheus.io/docs/visualization/browser/).

In the expression box, type in a metric name (for example, `rdi_engine_state`) and select `Enter` or the `Execute` button to see the following result:

[code example]

> Note: You may see more than just one Write-behind metric if Write-behind engine has already processed some data. If you do not see any metrics, check your scraper job configuration in Prometheus.

title: Use Grafana to analyze metrics

text:
Optionally, you may deploy the sample Grafana dashboard to monitor the status of Write-behind engine and its individual jobs. Follow these steps to import the dashboard:

1. Download the **dashboard file** to your local machine.

1. Log into Grafana and navigate to the list of dashboards, then choose **New > Import**:

1. On the next screen, select **Upload JSON file** and upload the file you downloaded in step 1. Make sure you select the data source that is connected to the Write-behind metrics exporter:

1. Select **Import** and make sure you choose the jobs to monitor from the drop-down list (this will be empty if you don't have any jobs running yet):

title: Write-behind metrics

text:
This list shows exported Write-behind metrics along with their descriptions:

| Metric Name                 | Labels                                                                                | Values               | Description                                                                                |
| --------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| rdi_engine_state            | {status=RUNNING \| STOPPED, sync_mode=SNAPSHOT \| STREAMING \| UNKNOWN}                                    | 0, 1                 | Status of Write-behind engine. 0 - Write-behind engine is stopped, 1 - Write-behind engine is running. Sync mode label indicates the last reported ingest synchronization mode.                                                               |
| rdi_incoming_entries        | {data_source=`<stream name>`, operation=pending \| inserted \| updated \| deleted \| filtered \| rejected} | `<count of records>` | Counters, indicating the number of operations performed for each stream.                                                                                |
| rdi_stream_event_latency_ms | {data_source=`<stream name>`}                                                                              | 0 - ∞          | Latency calculated for each stream. Indicates the time in milliseconds the first available record has spent in the stream waiting to be processed by Write-behind engine. If no records pending it will always return zero. |

code: redis-di monitor

section_id: test-write-behind-metrics-exporter

code:
scrape_configs:
  # scrape Write-behind metrics exporter
  - job_name: rdi-exporter
    static_configs:
      - targets: ["redis-exporter:9121"]

section_id: configure-prometheus

code: rdi_engine_state{instance="redis-exporter:9121", job="rdi-exporter", status="RUNNING", sync_mode="UNKNOWN"} 1

section_id: test-the-prometheus-scraper


title: Quickstart

summary: Get started creating a write-behind pipeline

title: Overview

text: This guide takes you through the creation of a write-behind pipeline.

title: Concepts

text:
**Write-behind** is a processing pipeline used to synchronize data in a Redis database with a downstream data store.
You can think about it as a pipeline that starts with change data capture (CDC) events for a Redis database and then filters, transforms, and maps the data to the target data store data structures.

The **target** data store to which the write-behind pipeline connects and writes data.

The write-behind pipeline is composed of one or more **jobs**. Each job is responsible for capturing change for one key pattern in Redis and mapping it to one or more tables in the downstream data store. Each job is defined in a YAML file.

title: Supported data stores

role: compatibility

text:
Write-behind currently supports these target data stores:

| Data Store |
| ---------- |
| Cassandra  |
| MariaDB    |
| MySQL      |
| Oracle     |
| PostgreSQL |
| Redis Enterprise     |
| SQL Server |

title: Prerequisites

text:
The only prerequisite for running Write-behind is [Redis Gears Python](https://redis.com/modules/redis-gears/) >= 1.2.6 installed on the Redis Enterprise Cluster and enabled for the database you want to mirror to the downstream data store.
For more information, see
[RedisGears installation]().

title: Preparing the write-behind pipeline

text:
- Install [Write-behind CLI]() on a Linux host that has connectivity to your Redis Enterprise Cluster.
- Run the [`configure`]() command to install the Write-behind Engine on your Redis database, if you have not used this Redis database with Write-behind before.
- Run the [`scaffold`]() command with the type of data store you want to use, for example:

  [code example]

  This creates a template `config.yaml` file and a folder named `jobs` under the current directory.
  You can specify any folder name with `--dir` or use the `--preview config.yaml` option, if your Write-behind CLI is deployed inside a Kubernetes (K8s) pod, to get the `config.yaml` template to the terminal.

- Add the connections required for downstream targets in the `connections` section of `config.yaml`, for example:

  [code example]

  This is the first section of the `config.yaml` file and typically the only one you'll need to edit. The `connections` section is designed to have many target connections. In the previous example, there are two downstream connections named `my-postgres` and `my-mysql`.

  To obtain a secured connection using TLS, you can add more `connect_args` or `query_args`, depending on the specific target database terminology, to the connection definition.

  The name can be any arbitrary name as long as it is:

  - Unique for this Write-behind engine
  - Referenced correctly by the jobs in the respective YAML files

In order to prepare the pipeline, fill in the correct information for the target data store. Secrets can be provided using a reference to a secret ([see below](#how-to-provide-targets-secrets)) or by specifying a path.

The `applier` section has information about the batch size and frequency used to write data to the target.

Some of the `applier` attributes such as `target_data_type`, `wait_enabled`, and `retry_on_replica_failure` are specific for the Write-behind ingest pipeline and can be ignored.

title: Write-behind jobs

text:
Write-behind jobs are a mandatory part of the write-behind pipeline configuration.
Under the `jobs` directory (parallel to `config.yaml`) you should have a job definition in a YAML file for every key pattern you want to write to a downstream database table.

The YAML file can be named using the destination table name or another naming convention, but it has to have a unique name.

Job definition has the following structure:

[code example]

title: Source section

text:
The `source` section describes the source of data in the pipeline.

The `redis` section is common for every pipeline initiated by an event in Redis, such as applying changes to data. In the case of write-behind, it has the information required to activate a pipeline dealing with changes to data. It includes the following attributes:

- The `key_pattern` attribute specifies the pattern of Redis keys to listen on. The pattern must correspond to keys that are of Hash or JSON type.

- The `exclude_commands` attribute specifies which commands to ignore. For example, if you listen on a key pattern with Hash values, you can exclude the `HDEL` command so no data deletions will propagate to the downstream database. If you don't specify this attribute, Write-behind acts on all relevant commands.
- The `trigger` attribute is mandatory and must be set to `write-behind`.

- The `row_format` attribute can be used with the value `full` to receive both the `before` and `after` sections of the payload. Note that for write-behind events the `before` value of the key is never provided.

 > Note: Write-behind does not support the [`expired`](#events-generated-by-different-commands) event. Therefore, keys that are expired in Redis will not be deleted from the target database automatically.
> Notes: The `redis` attribute is a breaking change replacing the `keyspace` attribute. The `key_pattern` attribute replaces the `pattern` attribute. The `exclude_commands` attributes replaces the `exclude-commands` attribute. If you upgrade to version 0.105 and beyond, you must edit your existing jobs and redeploy them.

title: Output section

role: returns

text:
The `output` section is critical. It specifies a reference to a connection from the `config.yaml` `connections` section:

- The `uses` attribute specifies the type of **writer** Write-behind will use to prepare and write the data to the target.
  In this example, it is `relational.write`, a writer that translates the data into a SQL statement with the specific dialect of the downstream relational database.
  For a full list of supported writers, see
  [data transformation block types]().

- The `schema` attribute specifies the schema/database to use (different database have different names for schema in the object hierarchy).

- The `table` attribute specifies the downstream table to use.

- The `keys` section specifies the field(s) in the table that are the unique constraints in that table.

- The `mapping` section is used to map database columns to Redis fields with different names or to expressions. The mapping can be all Redis data fields or a subset of them.

> Note: The columns used in `keys` will be automatically included, so there's no need to repeat them in the `mapping` section.

title: Apply filters and transformations to write-behind

text:
The Write-behind jobs can apply filters and transformations to the data before it is written to the target. Specify the filters and transformations under the `transform` section.

#### Filters

Use filters to skip some of the data and not apply it to target.
Filters can apply simple or complex expressions that take as arguments the Redis entry key, fields, and even the change op code (create, delete, update, etc.).
See [Filter]() for more information.

#### Transformations

Transformations manipulate the data in one of the following ways:

- Renaming a field
- Adding a field
- Removing a field
- Mapping source fields to use in output

To learn more about transformations, see
[data transformation pipeline]().

title: Provide target's secrets

text: The target's secrets (such as TLS certificates) can be read from a path on the Redis node's file system. This allows the consumption of secrets injected from secret stores.

title: Deploy the write-behind pipeline

text:
To start the pipeline, run the
[`deploy`]() command:

[code example]

You can check that the pipeline is running, receiving, and writing data using the
[`status`]() command:

[code example]

title: Monitor the write-behind pipeline

text:
The Write-behind pipeline collects the following metrics:

| Metric Description                  | Metric in [Prometheus](https://prometheus.io/)                                                      |
| ----------------------------------- | --------------------------------------------------------------------------------------------------- |
| Total incoming events by stream     | Calculated as a Prometheus DB query: `sum(pending, rejected, filtered, inserted, updated, deleted)` |
| Created incoming events by stream   | `rdi_metrics_incoming_entries{data_source:"…",operation="inserted"}`                                |
| Updated incoming events by stream   | `rdi_metrics_incoming_entries{data_source:"…",operation="updated"}`                                 |
| Deleted incoming events by stream   | `rdi_metrics_incoming_entries{data_source:"…",operation="deleted"}`                                 |
| Filtered incoming events by stream  | `rdi_metrics_incoming_entries{data_source:"…",operation="filtered"}`                                |
| Malformed incoming events by stream | `rdi_metrics_incoming_entries{data_source:"…",operation="rejected"}`                                |
| Total events per stream (snapshot)  | `rdi_metrics_stream_size{data_source:""}`                                                           |
| Time in stream (snapshot)           | `rdi_metrics_stream_last_latency_ms{data_source:"…"}`                                               |

To use the metrics you can either:

- Run the [`status`]() command:

  [code example]

- Scrape the metrics using Write-behind's Prometheus exporter

title: Upgrading

text:
If you need to upgrade Write-behind, you should use the
[`upgrade`]() command that provides for a zero downtime upgrade:

[code example]

code: redis-di scaffold --strategy write_behind --dir . --db-type mysql

section_id: preparing-the-write-behind-pipeline

code:
connections:
    my-postgres:
      type: postgresql
      host: 172.17.0.3
      port: 5432
      database: postgres
      user: postgres
      password: postgres
      #query_args:
      # sslmode: verify-ca
      # sslrootcert: /opt/work/ssl/ca.crt
      # sslkey: /opt/work/ssl/client.key
      # sslcert: /opt/work/ssl/client.crt
    my-mysql:
      type: mysql
      host: 172.17.0.4
      port: 3306
      database: test
      user: test
      password: test
      #connect_args:
      # ssl_ca: /opt/ssl/ca.crt
      # ssl_cert: /opt/ssl/client.crt
      # ssl_key: /opt/ssl/client.key

code:
source:
  redis:
    key_pattern: emp:*
    trigger: write-behind
    exclude_commands: ["json.del"]
transform:
  - uses: rename_field
    with:
      from_field: after.country
      to_field: after.my_country
output:
  - uses: relational.write
    with:
      connection: my-connection
      schema: my-schema
      table: my-table
      keys:
        - first_name
        - last_name
      mapping:
        - first_name
        - last_name
        - address
        - gender

section_id: write-behind-jobs

code: redis-di deploy

section_id: deploy-the-write-behind-pipeline

code: redis-di status

section_id: monitor-the-write-behind-pipeline

code: redis-di upgrade ...

section_id: upgrading


title: CLI

summary: Write-behind CLI reference

summary: A command line tool to manage & configure Write-behind

title: redis-di

summary: Adds a new context

title: redis-di add-context

summary: Configures the Write-behind Database so it is ready to process data

title: redis-di configure

summary: Creates the Write-behind Database instance

title: redis-di create

summary: Deletes Write-behind database permanently

title: redis-di delete

summary: Deletes all contexts

title: redis-di delete-all-contexts

summary: Deletes a context

title: redis-di delete-context

summary: Deploys the Write-behind configurations including target

title: redis-di deploy

summary: Describes a transformation engine's job

title: redis-di describe-job

summary: Dumps Write-behind support package

title: redis-di dump-support-package

summary: Returns all the stored rejected entries

title: redis-di get-rejected

summary: Lists all saved contexts

title: redis-di list-contexts

summary: Lists transformation engine's jobs

title: redis-di list-jobs

summary: Monitors Write-behind by collecting metrics and exporting to Prometheus

title: redis-di monitor

summary: Resets the pipeline into initial full sync mode

title: redis-di reset

summary: Generates configuration files for Write-behind and Debezium (when ingesting data to Redis)

title: redis-di scaffold

summary: Sets a context to be the active one

title: redis-di set-context

summary: Writes a secret to Redis secret store

title: redis-di set-secret

summary: Starts the pipeline

title: redis-di start

summary: Displays the status of the pipeline end to end

title: redis-di status

summary: Stops the pipeline

title: redis-di stop

summary: Starts a trace session for troubleshooting data transformation

title: redis-di trace

summary: Upgrades Write-behind Engine without losing data or downtime

title: redis-di upgrade


title: redis-di

title: Usage

role: syntax

text: [code example]

title: Options

role: parameters

text:
- `version`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--version`

  Show the version and exit.

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

section_id: usage

code:
Usage: redis-di [OPTIONS] COMMAND [ARGS]...

  A command line tool to manage & configure Write-behind

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  add-context           Adds a new context
  configure             Configures the Write-behind Database so it is ready to...
  create                Creates the Write-behind Database instance
  delete                Deletes Write-behind database permanently
  delete-all-contexts   Deletes all contexts
  delete-context        Deletes a context
  deploy                Deploys the Write-behind configurations including target
  describe-job          Describes a transformation engine's job
  dump-support-package  Dumps Write-behind support package
  get-rejected          Returns all the stored rejected entries
  list-contexts         Lists all saved contexts
  list-jobs             Lists transformation engine's jobs
  monitor               Monitors Write-behind by collecting metrics and exporting...
  reset                 Resets the pipeline into initial full sync mode
  scaffold              Generates configuration files for Write-behind and...
  set-context           Sets a context to be the active one
  set-secret            Writes a secret to Redis secret store
  start                 Starts the pipeline
  status                Displays the status of the pipeline end to end
  stop                  Stops the pipeline
  trace                 Starts a trace session for troubleshooting data...
  upgrade               Upgrades Write-behind Engine without losing data or downtime

section_id: cli-help


title: redis-di add-context

summary: Adds a new context

title: Usage

text: [code example]

title: Options

text:
- `context_name` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `context-name`

- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `cluster_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-host`

  Host/IP of Redis Enterprise Cluster (service name in case of k8s)

- `cluster_api_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9443`
  - Usage: `--cluster-api-port`

  API Port of Redis Enterprise Cluster

- `cluster_user` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-user`

  Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di add-context [OPTIONS] CONTEXT_NAME

  Adds a new context

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --cluster-host TEXT             Host/IP of Redis Enterprise Cluster (service
                                  name in case of k8s)  [required]
  --cluster-api-port INTEGER RANGE
                                  API Port of Redis Enterprise Cluster
                                  [default: 9443; 1000<=x<=65535; required]
  --cluster-user TEXT             Redis Enterprise Cluster username with
                                  either DB Member, Cluster Member or Cluster
                                  Admin roles  [required]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --help                          Show this message and exit.


title: redis-di configure

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di configure [OPTIONS]

code:
Usage: redis-di configure [OPTIONS]

  Configures the Write-behind Database so it is ready to process data

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --help                          Show this message and exit.


title: redis-di create

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `silent`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--silent`

  Silent install. Do not prompt to enter missing parameters

- `no_configure`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--no-configure`

  Do not install Write-behind Engine to the Write-behind Database

- `cluster_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-host`

  Host/IP of Redis Enterprise Cluster (service name in case of k8s)

- `cluster_api_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9443`
  - Usage: `--cluster-api-port`

  API Port of Redis Enterprise Cluster

- `cluster_user` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-user`

  Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles

- `cluster_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-password`

  Redis Enterprise Cluster Password

- `rdi_port`:

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port for the new Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: ``
  - Usage: `--rdi-password`

  Password for the new Write-behind Database (alphanumeric characters with zero or more of the following: ! & # $ ^ < > -)

- `rdi_memory`:

  - Type: <IntRange x>=30>
  - Default: `100`
  - Usage: `--rdi-memory`

  Memory for Write-behind Database (in MB)

- `rdi_shards`:

  - Type: <IntRange x>=1>
  - Default: `1`
  - Usage: `--rdi-shards`

  Number of database server-side shards

- `replication`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--replication`

  In-memory database replication

- `redisgears_module`:

  - Type: STRING
  - Default: ``
  - Usage: `--redisgears-module`

  RedisGears module file

- `with_rejson`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--with-rejson`

  Include ReJSON in the Write-behind Database

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di create [OPTIONS]

code:
Usage: redis-di create [OPTIONS]

  Creates the Write-behind Database instance

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --silent                        Silent install. Do not prompt to enter
                                  missing parameters
  --no-configure                  Do not install Write-behind Engine to the Write-behind
                                  Database
  --cluster-host TEXT             Host/IP of Redis Enterprise Cluster (service
                                  name in case of k8s)  [required]
  --cluster-api-port INTEGER RANGE
                                  API Port of Redis Enterprise Cluster
                                  [default: 9443; 1000<=x<=65535; required]
  --cluster-user TEXT             Redis Enterprise Cluster username with
                                  either DB Member, Cluster Member or Cluster
                                  Admin roles  [required]
  --cluster-password TEXT         Redis Enterprise Cluster Password
  --rdi-port INTEGER RANGE        Port for the new Write-behind Database
                                  [1000<=x<=65535]
  --rdi-password TEXT             Password for the new Write-behind Database
                                  (alphanumeric characters with zero or more
                                  of the following: ! & # $ ^ < > -)
  --rdi-memory INTEGER RANGE      Memory for Write-behind Database (in MB)  [x>=30]
  --rdi-shards INTEGER RANGE      Number of database server-side shards
                                  [x>=1]
  --replication                   In-memory database replication
  --redisgears-module TEXT        RedisGears module file
  --help                          Show this message and exit.


title: redis-di delete

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `cluster_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-host`

  Host/IP of Redis Enterprise Cluster (service name in case of k8s)

- `cluster_api_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9443`
  - Usage: `--cluster-api-port`

  API Port of Redis Enterprise Cluster

- `cluster_user` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-user`

  Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles

- `cluster_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-password`

  Redis Enterprise Cluster Password

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `force`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--force
-f`

  Force operation. skips verification prompts

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di delete [OPTIONS]

code:
Usage: redis-di delete [OPTIONS]

  Deletes Write-behind database permanently

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --cluster-host TEXT             Host/IP of Redis Enterprise Cluster (service
                                  name in case of k8s)  [required]
  --cluster-api-port INTEGER RANGE
                                  API Port of Redis Enterprise Cluster
                                  [default: 9443; 1000<=x<=65535; required]
  --cluster-user TEXT             Redis Enterprise Cluster username with
                                  either DB Member, Cluster Member or Cluster
                                  Admin roles  [required]
  --cluster-password TEXT         Redis Enterprise Cluster Password
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  -f, --force                     Force operation. skips verification prompts
  --help                          Show this message and exit.


title: redis-di delete-all-contexts

summary: Deletes all contexts

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `force`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--force
-f`

  Force operation. skips verification prompts

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di delete-all-contexts [OPTIONS]

  Deletes all contexts

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  -f, --force                     Force operation. skips verification prompts
  --help                          Show this message and exit.


title: redis-di delete-context

summary: Deletes a context

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `context_name` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `context-name`

- `force`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--force
-f`

  Force operation. skips verification prompts

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di delete-context [OPTIONS] CONTEXT_NAME

  Deletes a context

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  -f, --force                     Force operation. skips verification prompts
  --help                          Show this message and exit.


title: redis-di deploy

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `directory`:

  - Type: STRING
  - Default: `.`
  - Usage: `--dir`

  Directory containing Write-behind configuration

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di deploy [OPTIONS]

code:
Usage: redis-di deploy [OPTIONS]

  Deploys the Write-behind configurations including target

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --dir TEXT                      Directory containing Write-behind configuration
                                  [default: .]
  --help                          Show this message and exit.


title: redis-di describe-job

summary: Describes a transformation engine's job

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `job_name` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `job-name`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di describe-job [OPTIONS] JOB_NAME

  Describes a transformation engine's job

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --help                          Show this message and exit.


title: redis-di dump-support-package

summary: Dumps Write-behind support package

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `directory`:

  - Type: STRING
  - Default: `.`
  - Usage: `--dir`

  Directory where the support file should be generated

- `dump_rejected`:

  - Type: INT
  - Default: `none`
  - Usage: `--dump-rejected`

  Dumps rejected records

- `trace_timeout`:

  - Type: <IntRange 1<=x<=600>
  - Default: `none`
  - Usage: `--trace-timeout`

  Stops the trace after exceeding this timeout (in seconds)

- `max_change_records`:

  - Type: <IntRange x>=1>
  - Default: `10`
  - Usage: `--max-change-records`

  Maximum traced change records per shard

- `trace_only_rejected`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--trace-only-rejected`

  Trace only rejected change records

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di dump-support-package [OPTIONS]

  Dumps Write-behind support package

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --dir TEXT                      Directory where the support file should be
                                  generated  [default: .]
  --dump-rejected INTEGER         Dumps rejected records
  --trace-timeout INTEGER RANGE   Stops the trace after exceeding this timeout
                                  (in seconds)  [1<=x<=600]
  --max-change-records INTEGER RANGE
                                  Maximum traced change records per shard
                                  [x>=1]
  --trace-only-rejected           Trace only rejected change records
  --help                          Show this message and exit.


title: redis-di get-rejected

summary: Returns all the stored rejected entries

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `max_records`:

  - Type: <IntRange x>=1>
  - Default: `none`
  - Usage: `--max-records`

  Maximum rejected records per DLQ

- `oldest`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--oldest
-o`

  Displays the oldest rejected records. If omitted, most resent records will be retrieved

- `dlq_name`:

  - Type: STRING
  - Default: `none`
  - Usage: `--dlq-name`

  Only prints the rejected records for the specified DLQ (Dead Letter Queue) name

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di get-rejected [OPTIONS]

code:
Usage: redis-di get-rejected [OPTIONS]

  Returns all the stored rejected entries

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --max-records INTEGER RANGE     Maximum rejected records per DLQ  [x>=1]
  -o, --oldest                    Displays the oldest rejected records. If
                                  omitted, most resent records will be
                                  retrieved
  --dlq-name TEXT                 Only prints the rejected records for the
                                  specified DLQ (Dead Letter Queue) name
  --help                          Show this message and exit.


title: redis-di list-contexts

summary: Lists all saved contexts

title: Usage

text: [code example]

title: Options

text:
- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di list-contexts [OPTIONS]

code:
Usage: redis-di list-contexts [OPTIONS]

  Lists all saved contexts

Options:
  --help  Show this message and exit.


title: redis-di list-jobs

summary: Lists transformation engine's jobs

title: Usage

text: [code example]

title: Options

title: CLI help

text: [code example]

code: Usage: redis-di list-jobs [OPTIONS]

code:
Usage: redis-di list-jobs [OPTIONS]

  Lists transformation engine's jobs

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --help                          Show this message and exit.


title: redis-di monitor

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `exporter_port`:

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9121`
  - Usage: `--exporter-port`

  HTTP port to start the exporter on

- `collect_interval`:

  - Type: <IntRange 1<=x<=60>
  - Default: `5`
  - Usage: `--collect-interval`

  Metrics collection interval (seconds)

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di monitor [OPTIONS]

code:
Usage: redis-di monitor [OPTIONS]

  Monitors Write-behind by collecting metrics and exporting to Prometheus

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --exporter-port INTEGER RANGE   HTTP port to start the exporter on
                                  [1000<=x<=65535]
  --collect-interval INTEGER RANGE
                                  Metrics collection interval (seconds)
                                  [1<=x<=60]
  --help                          Show this message and exit.


title: redis-di reset

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `force`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--force
-f`

  Force operation. skips verification prompts

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di reset [OPTIONS]

code:
Usage: redis-di reset [OPTIONS]

  Resets the pipeline into initial full sync mode

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  -f, --force                     Force operation. skips verification prompts
  --help                          Show this message and exit.


title: redis-di scaffold

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `db_type` (REQUIRED):

  - Type: Choice([<DbType.MYSQL: 'mysql'>, <DbType.ORACLE: 'oracle'>, <DbType.POSTGRESQL: 'postgresql'>, <DbType.REDIS: 'redis'>, <DbType.SQLSERVER: 'sqlserver'>])
  - Default: `none`
  - Usage: `--db-type`

  DB type

- `strategy`:

  - Type: Choice([<Strategy.INGEST: 'ingest'>, <Strategy.WRITE_BEHIND: 'write_behind'>])
  - Default: `ingest`
  - Usage: `--strategy`

  Strategy

  Output to directory or stdout

- `directory`:

  - Type: STRING
  - Default: `none`
  - Usage: `--dir`

  Directory containing Write-behind configuration

- `preview`:

  - Type: Choice(['debezium/application.properties', 'config.yaml'])
  - Default: `none`
  - Usage: `--preview`

  Print the content of specified config file to CLI output

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di scaffold [OPTIONS]

code:
Usage: redis-di scaffold [OPTIONS]

  Generates configuration files for Write-behind and Debezium (when ingesting data to
  Redis)

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --db-type [mysql|oracle|postgresql|redis|sqlserver]
                                  DB type  [required]
  --strategy [ingest|write_behind]
                                  Strategy  [default: ingest]
  Output formats: [mutually_exclusive, required]
                                  Output to directory or stdout
    --dir TEXT                    Directory containing Write-behind configuration
    --preview [debezium/application.properties|config.yaml]
                                  Print the content of specified config file
                                  to CLI output
  --help                          Show this message and exit.


title: redis-di set-context

summary: Sets a context to be the active one

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `context_name` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `context-name`

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code:
Usage: redis-di set-context [OPTIONS] CONTEXT_NAME

  Sets a context to be the active one

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --help                          Show this message and exit.


title: redis-di set-secret

summary: Writes a secret to Redis secret store

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `cluster_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-host`

  Host/IP of Redis Enterprise Cluster (service name in case of k8s)

- `cluster_api_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9443`
  - Usage: `--cluster-api-port`

  API Port of Redis Enterprise Cluster

- `cluster_user` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-user`

  Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles

- `cluster_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-password`

  Redis Enterprise Cluster Password

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `secret_name` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--secret-name`

  The name of the secret

- `secret_value` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--secret-value`

  The value of the secret

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di set-secret [OPTIONS]

code:
Usage: redis-di set-secret [OPTIONS]

  Writes a secret to Redis secret store

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --cluster-host TEXT             Host/IP of Redis Enterprise Cluster (service
                                  name in case of k8s)  [required]
  --cluster-api-port INTEGER RANGE
                                  API Port of Redis Enterprise Cluster
                                  [default: 9443; 1000<=x<=65535; required]
  --cluster-user TEXT             Redis Enterprise Cluster username with
                                  either DB Member, Cluster Member or Cluster
                                  Admin roles  [required]
  --cluster-password TEXT         Redis Enterprise Cluster Password
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --secret-name TEXT              The name of the secret  [required]
  --secret-value TEXT             The value of the secret  [required]
  --help                          Show this message and exit.


title: redis-di start

summary: Starts the pipeline

title: Usage

text: [code example]

title: Options

title: CLI help

text: [code example]

code: Usage: redis-di start [OPTIONS]

code:
Usage: redis-di start [OPTIONS]

  Starts the pipeline

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --help                          Show this message and exit.


title: redis-di status

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `live`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--live
-l`

  Live data flow monitoring

- `page_number`:

  - Type: <IntRange x>=1>
  - Default: `none`
  - Usage: `--page-number
-p`

  Set the page number (live mode only)

- `page_size`:

  - Type: <IntRange x>=1>
  - Default: `none`
  - Usage: `--page-size
-s`

  Set the page size (live mode only)

- `ingested_only`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--ingested-only
-i`

  Display ingested data streams (live mode only)

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di status [OPTIONS]

code:
Usage: redis-di status [OPTIONS]

  Displays the status of the pipeline end to end

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  -l, --live                      Live data flow monitoring
  -p, --page-number INTEGER RANGE
                                  Set the page number (live mode only)  [x>=1]
  -s, --page-size INTEGER RANGE   Set the page size (live mode only)  [x>=1]
  -i, --ingested-only             Display ingested data streams (live mode
                                  only)
  --help                          Show this message and exit.


title: redis-di stop

summary: Stops the pipeline

title: Usage

text: [code example]

title: Options

title: CLI help

text: [code example]

code: Usage: redis-di stop [OPTIONS]

code:
Usage: redis-di stop [OPTIONS]

  Stops the pipeline

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --help                          Show this message and exit.


title: redis-di trace

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `max_change_records`:

  - Type: <IntRange x>=1>
  - Default: `10`
  - Usage: `--max-change-records`

  Maximum traced change records per shard

- `timeout` (REQUIRED):

  - Type: <IntRange 1<=x<=600>
  - Default: `20`
  - Usage: `--timeout`

  Stops the trace after exceeding this timeout (in seconds)

- `trace_only_rejected`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--trace-only-rejected`

  Trace only rejected change records

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di trace [OPTIONS]

code:
Usage: redis-di trace [OPTIONS]

  Starts a trace session for troubleshooting data transformation

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  --max-change-records INTEGER RANGE
                                  Maximum traced change records per shard
                                  [x>=1]
  --timeout INTEGER RANGE         Stops the trace after exceeding this timeout
                                  (in seconds)  [default: 20; 1<=x<=600;
                                  required]
  --trace-only-rejected           Trace only rejected change records
  --help                          Show this message and exit.


title: redis-di upgrade

title: Usage

text: [code example]

title: Options

text:
- `loglevel`:

  - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL'])
  - Default: `info`
  - Usage: `--loglevel
-log-level`

- `cluster_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-host`

  Host/IP of Redis Enterprise Cluster (service name in case of k8s)

- `cluster_api_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `9443`
  - Usage: `--cluster-api-port`

  API Port of Redis Enterprise Cluster

- `cluster_user` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-user`

  Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles

- `cluster_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--cluster-password`

  Redis Enterprise Cluster Password

- `rdi_host` (REQUIRED):

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-host`

  Host/IP of Write-behind Database

- `rdi_port` (REQUIRED):

  - Type: <IntRange 1000<=x<=65535>
  - Default: `none`
  - Usage: `--rdi-port`

  Port of Write-behind Database

- `rdi_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-password`

  Write-behind Database Password

- `rdi_key`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key`

  Private key file to authenticate with

- `rdi_cert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cert`

  Client certificate file to authenticate with

- `rdi_cacert`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-cacert`

  CA certificate file to verify with

- `rdi_key_password`:

  - Type: STRING
  - Default: `none`
  - Usage: `--rdi-key-password`

  Password for unlocking an encrypted private key

- `force`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--force
-f`

  Force upgrade/downgrade. skips verification prompts

- `help`:

  - Type: BOOL
  - Default: `false`
  - Usage: `--help`

  Show this message and exit.

title: CLI help

text: [code example]

code: Usage: redis-di upgrade [OPTIONS]

code:
Usage: redis-di upgrade [OPTIONS]

  Upgrades Write-behind Engine without losing data or downtime

Options:
  -log-level, --loglevel [DEBUG|INFO|WARN|ERROR|CRITICAL]
                                  [default: INFO]
  --cluster-host TEXT             Host/IP of Redis Enterprise Cluster (service
                                  name in case of k8s)  [required]
  --cluster-api-port INTEGER RANGE
                                  API Port of Redis Enterprise Cluster
                                  [default: 9443; 1000<=x<=65535; required]
  --cluster-user TEXT             Redis Enterprise Cluster username with
                                  either DB Member, Cluster Member or Cluster
                                  Admin roles  [required]
  --cluster-password TEXT         Redis Enterprise Cluster Password
  --rdi-host TEXT                 Host/IP of Write-behind Database  [required]
  --rdi-port INTEGER RANGE        Port of Write-behind Database  [1000<=x<=65535;
                                  required]
  --rdi-password TEXT             Write-behind Database Password
  --rdi-key TEXT                  Private key file to authenticate with
  --rdi-cert TEXT                 Client certificate file to authenticate with
  --rdi-cacert TEXT               CA certificate file to verify with
  --rdi-key-password TEXT         Password for unlocking an encrypted private
                                  key
  -f, --force                     Force upgrade/downgrade. skips verification
                                  prompts
  --help                          Show this message and exit.


title: Write-behind configuration file

summary: Write-behind configuration file reference

title: Write-behind Configuration File

text:
**Properties**

| Name                                                                                | Type             | Description | Required |
| ------------------------------------------------------------------------------------------ | ---------------- | ----------- | -------- |
| [**applier**](#applier)
(Configuration details of Write-behind Applier Gear) | `object`, `null` |             |          |
| [**connections**](#connections)                                                            | `object`         |             |          |

title: applier: Configuration details of Write-behind Applier Gear

text:
**Properties**

| Name                                                                                | Type                | Description                                                                                | Required |
| ------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | --- |
| **on_failed_retry_interval**
(Interval \(in seconds\) on which to perform retry on failure)                                      | `integer`, `string` | Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **read_batch_size**
(The batch size for reading data from source database)                                                       | `integer`, `string` | Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **debezium_lob_encoded_placeholder**
(Enable Debezium LOB placeholders)                                                          | `string`            | Default: `"X19kZWJleml1bV91bmF2YWlsYWJsZV92YWx1ZQ=="`
                                                                                |          |
| **dedup**
(Enable deduplication mechanism)                                                                                | `boolean`           | Default: `false`
                                                                                |          |
| **dedup_max_size**
(Max size of the deduplication set)                                                                           | `integer`           | Default: `1024`
Minimum: `1`
                                                                                |          |
| **duration**
(Time \(in ms\) after which data will be read from stream even if read_batch_size was not reached)                  | `integer`, `string` | Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **write_batch_size**
(The batch size for writing data to target Redis database\. Should be less or equal to the read_batch_size) | `integer`, `string` | Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **error_handling**
(Error handling strategy: ignore \- skip, dlq \- store rejected messages in a dead letter queue)              | `string`            | Default: `"dlq"`
Pattern: ``^\${.\*}$                                                                                | ignore   | dlq``
   |     |
| **dlq_max_messages**
(Dead letter queue max messages per stream)                                                                 | `integer`, `string` | Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **target_data_type**
(Target data type: hash/json \- RedisJSON module must be in use in the target DB)                           | `string`            | Default: `"hash"`
Pattern: ``^\${.\*}$                                                                                | hash     | json``
  |     |
| **json_update_strategy**
(Target update strategy: replace/merge \- RedisJSON module must be in use in the target DB)             | `string`            | (DEPRECATED)
Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy.
Default: `"replace"`
Pattern: ``^\${.\*}$ | replace  | merge``
 |     |
| **initial_sync_processes**
(Number of processes Write-behind Engine creates to process the initial sync with the source)                  | `integer`, `string` | Default: `4`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `32`
                                                                                |          |
| **wait_enabled**
(Checks if the data has been written to the replica shard of the target database)                                                      | `boolean`           | Default: `false`
                                                                                |          |
| **wait_timeout**
(Timeout in milliseconds when checking write to the replica shard of the target database)                                              | `integer`, `string` | Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
                                                                                |          |
| **retry_on_replica_failure**
(Ensures that the data has been written to the replica shard and keeps retrying if not)             | `boolean`           | Default: `true`
                                                                                |          |

**Additional Properties:** not allowed

title: connections: Connections

text:
**Properties (Pattern)**

| Name                     | Type | Description | Required |
| ------------------------ | ---- | ----------- | -------- |
| **\.\***                 |      |             |          |
| **additionalProperties** |      |             |          |