Data Integration on Docs

RDI on Redis Cloud quick start

Mon, 01 Jan 0001 00:00:00 +0000

The rdi-cloud-automation GitHub repository contains a Terraform script that quickly sets up a PostgreSQL source database on an EC2 instance and all required permissions and network setup to connect it to a Redis Cloud target database.

Note:

This guide is for demonstration purposes only. It is not recommended for production use.

Prerequisites

To follow this guide, you need to:

Create and manage Data Integration workspace

Mon, 01 Jan 0001 00:00:00 +0000

Before you can create your first Data Integration pipeline for a Redis Cloud subscription, you must first deploy the cloud infrastructure needed to host the pipeline and run the workers associated with the pipeline. In Redis Cloud, this is called a Workspace. Each Pro subscription can have one Data integration workspace. You only need to set up the workspace once - any pipelines you create for your subscription will run on the workspace until you delete it. You won't be charged for a workspace until you start running your Data Integration pipeline.

Prepare source database

Mon, 01 Jan 0001 00:00:00 +0000

Prepare source database

Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See Prerequisites to find a list of supported source databases and database versions.

Create data pipeline

Mon, 01 Jan 0001 00:00:00 +0000

After you have prepared your source database and connection information, and created the workspace for your pipeline, you can set up your new pipeline.

In the Redis Cloud console, go to your target database and select the Data Integration tab. You'll see your subscription's workspace. Select Add pipeline to add a pipeline to the workspace.

You can also go to the Pipelines page from the left-hand menu and select Add pipeline from your workspace. Or, you can go to the Data Integration tab of your subscription and select Add pipeline from your workspace.

View and edit data pipeline

Mon, 01 Jan 0001 00:00:00 +0000

To view or manage your data pipeline, select it from your workspace from the Data Integration page or from the Data Integration tab in your subscription or database.

The pipeline page has the following tabs:

The following sections describe each of these tabs, as well as any actions that you can perform.

Dashboard

The Dashboard tab shows an overview and high-level statistics for the data pipeline.

RDS Proxy setup for RDI

Mon, 01 Jan 0001 00:00:00 +0000

Warning:

We do not recommend using RDS Proxy for RDI connections. The Lambda function approach provides better failover handling and is the recommended solution for production environments.

Additionally, RDS Proxy does not work with RDS PostgreSQL and Aurora PostgreSQL because it does not support PostgreSQL logical replication.

Only use RDS Proxy if you have specific requirements that necessitate it.