Create and manage Data Integration workspace

Create and manage the infrastructure for your Data Integration pipelines.

Before you can create your first Data Integration pipeline for a Redis Cloud subscription, you must first deploy the cloud infrastructure needed to host the pipeline and run the workers associated with the pipeline. In Redis Cloud, this is called a Workspace. Each Pro subscription can have one Data integration workspace. You only need to set up the workspace once - any pipelines you create for your subscription will run on the workspace until you delete it. A workspace can run several pipelines. See Add pipelines to a workspace. You won't be charged for a workspace until you deploy your first Data Integration pipeline.

Create a Data Integration workspace

For multiple source databases or Redis subscriptions, see Plan source connectivity. It explains which endpoint services you can reuse and which connections you configure for each workspace.

To create a Data Integration workspace for an existing Pro subscription:

1. In the Redis Cloud console, select Data Integration.

2. Select Create workspace if you have no workspaces yet.

   

   If you already have a workspace, select New workspace.

   

   You can also open Data Integration from a subscription or database page. Select Create workspace to use that subscription.

   

3. Select your Pro subscription if it is not already selected.

   

4. Review Data Integration subnet (CIDR). The console suggests a dedicated /22 Classless Inter-Domain Routing (CIDR) range for the workspace.

   Plan network capacity for all pipelines before you create the workspace. Choose a /21 or larger range if you plan more than 5 sources or more than 5 processor replicas in total. These counts are a conservative planning trigger, not a capacity limit. You cannot enlarge the workspace CIDR after creation.

   For Amazon Web Services (AWS), the RDI workspace CIDR must:

   • Be in the same RFC 1918 private address range as the subscription virtual private cloud (VPC)'s primary CIDR: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
   • Not overlap with existing subscription, peering, transit gateway (TGW), application, database, or RDI workspace CIDR ranges.

   For example, a subscription VPC has the primary CIDR 10.238.252.0/24. The range 192.168.0.0/22 is invalid because it is in a different RFC 1918 range. An unused range such as 10.239.0.0/22 is valid.

   If the suggestion is missing or unsuitable, select another unused range. It must meet your capacity needs and use the same private address range. See VPC CIDR block association restrictions.

   

5. Select Create workspace.

   

Redis Cloud creates the workspace in the background. Select Create pipeline to create your pipeline while it provisions. Select Create pipeline later to return to the Redis Cloud console.

View workspace status and details

You can view your workspace in one of the following ways:

• From the Redis Cloud console, go to the Data integration page, or
• From your subscription, select the Data Integration tab

There, you'll see your workspace and its pipelines, one row per pipeline. The Status column shows Draft for a pipeline that you have not deployed yet, and the state of a deployed pipeline otherwise, such as Running, Stopped, or Error. The Sources column lists each pipeline's source names and database types. A source with unfinished configuration is marked Pending setup. Open its draft to continue setup. The Billing column shows Not billed for a draft, and the hourly rate of a deployed pipeline. A stopped pipeline is still billed and shows Billing continues. See How does stopping sources or the pipeline affect billing? To see your workspace details, including the deployment CIDR and region information, select Workspace actions > Workspace details.

Add pipelines to a workspace

A workspace can run several pipelines. Each pipeline has its own sources, target database, processor, and settings, so you can deploy, stop, reset, or delete one pipeline without affecting the others. Use separate pipelines when the data goes to different target databases, or when you want to manage some of the data independently. To capture several source databases into the same target as one unit, add sources to one pipeline instead.

To add a pipeline:

1. Open your workspace from the Data Integration page or from the Data Integration tab of your subscription or database.

2. Select Add pipeline at the bottom of the workspace.

3. Enter a Pipeline name. Use 3 to 20 characters: lowercase letters, numbers, and hyphens. Start with a lowercase letter, end with a letter or number, and don't use two hyphens in a row. The name rdi is reserved, and each pipeline in the workspace must have a different name. You can't change the name later.

4. Select Create pipeline.

The new pipeline appears in the workspace with the Draft status, next to the other pipelines of the workspace, like inventory-propagation in the example in View workspace status and details. Redis Cloud doesn't bill a pipeline until you deploy it.

Continue with Create data pipeline to configure the pipeline's settings and sources and deploy it. You can set up one pipeline at a time. Until you deploy the new pipeline or delete its draft, Add pipeline is unavailable, and the console shows A new pipeline can be added only when the current pending pipeline setup is complete.

The pipelines of a workspace share its resources:

• The workspace CIDR range. Plan it for all the sources and processor replicas of all your pipelines.
• The workspace's AWS PrivateLink connections. A pipeline can use an endpoint service that another pipeline of the workspace already uses. See Plan source connectivity.
• The Role ARNs for source connectivity and secrets, which are the same for every pipeline of the workspace.
• The workspace's RDI database, which holds the change records of every pipeline until its processor writes them to the target.

Each pipeline is billed for its own collectors and processor replicas, and the workspace charge applies once, however many pipelines you deploy. See How is RDI usage calculated?

Delete workspace

When you delete the last pipeline of a workspace, you can delete the workspace in the same step. See Delete pipeline.

To delete your workspace, select Workspace actions > Delete workspace from your workspace.