Redis Data Integration configuration file

Reference for the RDI config.yaml file

This document describes the RDI config.yaml file in detail. See Configure data pipelines for more information about the role config.yaml plays in defining a pipeline.

Note about fully-qualified table names

Throughout this document we use the format <databaseName>.<tableName> to refer to a fully-qualified table name. This format is actually the one used by MySQL and MariaDB, but for Oracle, SQL Server, and PostgreSQL, you should use <schemaName>.<tableName> instead.

The example below shows the MySQL format specifying the desired columns for the chinook.customer and chinook.employee tables:

tables:
   chinook.customer:
       columns:
           - CustID
           - FirstName
           - LastName
           - Company
           - Address
           - Email
   chinook.employee:
       columns:
           - EmpID
           - FirstName
           - LastName
           - ReportsTo
           - Address
           - City
           - State

Top level objects

These objects define the sections at the root level of config.yaml.

Properties

sources: Source collectors

Each source database type has its own connector, but the basic configuration properties are the same for all databases.

See the Debezium documentation for more information about the specific connectors:

• MySQL/MariaDB
• Oracle
• PostgreSQL
• SQL Server

Essential properties

connection

Advanced properties

sink

source

Using custom queries in the initial snapshot

By default, the initial snapshot captures all rows from each table. If you want the snapshot to include only a subset of the rows in a table, you can use a custom SELECT statement to override the default and select only the rows you are interested in. To do this, you must first specify the tables whose SELECT statement you want to override by adding a snapshot.select.statement.overrides in the source section with a comma-separated list of fully-qualified table names.

After the snapshot.select.statement.overrides list, you must then add another configuration property for each table in the list to specify the custom SELECT statement for that table. The format of the property name depends on the database you are using:

• For Oracle, SQL Server, and PostrgreSQL, use snapshot.select.statement.overrides.<SCHEMA_NAME>.<TABLE_NAME>
• For MySQL and MariaDB, use: snapshot.select.statement.overrides<DATABASE_NAME>.<TABLE_NAME>

For example, with PostgreSQL, you would have a configuration like the following:

source:
    snapshot.select.statement.overrides: myschema.mytable
    snapshot.select.statement.overrides.myschema.mytable: |
        SELECT ...

For MySQL, you would have:

source:
    snapshot.select.statement.overrides: mydatabase.mytable
    snapshot.select.statement.overrides.mydatabase.mytable: |
        SELECT ...

You must also add the list of columns you want to include in the custom SELECT statement using fully-qualified names under "sources.tables". Specify each column in the configuration as shown below:

tables:
    # For MySQL and MariaDB: use database_name.table_name
    schema_name.table_name:
        columns:
            - column_name1 # Each column on a new line
            - column_name2
            - column_name3

If you want to capture all columns from a table, you can use empty curly braces {} instead of listing all the individual columns:

tables:
   # Captures all columns. For MySQL and MariaDB: use database_name.table_name.
   schema_name.table_name: {}

The example configuration below selects the columns CustomerId, FirstName and LastName from the customer table and joins it with the invoice table to select customers with total invoices greater than 8000:

tables:
 chinook.customer:
    columns:
        - CustomerID
        - FirstName
        - LastName

advanced:
    source:
        snapshot.select.statement.overrides: chinook.customer
        snapshot.select.statement.overrides.chinook.customer: |
            SELECT c.CustomerId, c.FirstName, c.LastName
            FROM chinook.customer c
            INNER JOIN chinook.invoice inv
            ON c.CustomerId = inv.CustomerId
            WHERE inv.total > 8000

Specifying custom message keys for change event records

By default, Debezium uses the primary key column(s) of a table as the message key for the records that it emits. However, you might want to configure custom message keys based on one or more columns to override this default behavior, or to specify message keys for tables that don't have a primary key.

Use the sources.tables section of config.yaml to specify a custom message key for one or more tables. List the columns you want to capture from the table under columns and list the columns you want to use for the message key under keys, as shown below:

# To include entries for multiple tables, simply add each table with its corresponding columns and keys under the 'tables' field.
tables:
   <DATABASE_NAME>.<TABLE_NAME>:
       columns:
           - <COLUMN(S)_LIST> # List of columns to include
       keys:
           - <COLUMN(S)_LIST> # Column(s) to be used together as the primary key

Note that you must add the columns you use for the message key to both the keys list and the columns list. You can use as many columns as you like to create the custom message keys but we recommend you use only the minimum set of columns required to guarantee a unique key for each message.

processors: RDI processors

Properties

Additional properties

Not allowed

targets: Target connections

Properties

targets.connection: Connection details

Properties