{
  "schema_version": 2,
  "id": "operate/rc/rdi/scale-pipeline",
  "title": "Scale a Cloud RDI pipeline",
  "url": "https://redis.io/docs/latest/operate/rc/rdi/scale-pipeline/",
  "summary": "Increase collector, RDI database, and processor capacity for a Cloud RDI pipeline.",
  "aliases": [
    "/operate/rc/rdi/scale-processor/",
    "/operate/rc/rdi/scale-processor",
    "/operate/rc/databases/rdi/scale-processor/",
    "/operate/rc/databases/rdi/scale-processor"
  ],
  "tags": [
    "docs",
    "operate",
    "rc"
  ],
  "last_updated": "2026-10-21T08:57:20+03:00",
  "page_type": "content",
  "content_hash": "e6b1c86dbc1e5dc63c42f48b5a2fc0c2962b3bc39a7bb42b9e4f8dd4b04dabd2",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "Every Redis Data Integration (RDI) pipeline in Redis Cloud uses the Flink\nprocessor. The collector, RDI database, processor, or target database can\nlimit pipeline capacity. Identify the bottleneck before you change a setting.\n\nCloud RDI does not automatically add or remove processor replicas\n(TaskManagers) based on load, pending records, throughput, or backpressure.\nSet the number of processor replicas when you need more processing capacity."
    },
    {
      "id": "plan-workspace-network-capacity",
      "title": "Plan workspace network capacity",
      "role": "content",
      "text": "Choose the workspace Classless Inter-Domain Routing (CIDR) range for all\npipelines you plan to run. All sources and processor replicas share the\nworkspace's network capacity. Each source uses a collector. Each TaskManager\nis a processor replica. Adding either uses more workspace addresses.\n\nThe console suggests a `/22` range. If you expect more than 5 sources or more\nthan 5 processor replicas in total across the workspace, choose a `/21` or\nlarger range when you create it. For further growth, consider a `/20` or larger\nrange. A `/21` has twice the address space of a `/22`; a `/20` has twice that\nof a `/21`. Doubling the address space does not guarantee twice as many\nsources or processor replicas.\n\nThis is conservative planning guidance, not a guaranteed capacity limit.\nLeave room for planned growth, replacement resources, and maintenance.\nThe range must also meet the requirements of your connectivity method and\nmust not overlap with connected networks."
    },
    {
      "id": "processor-capacity-by-cidr",
      "title": "Processor capacity by CIDR",
      "role": "content",
      "text": "Use this table as a reference for processor address capacity. The reference\nconfiguration uses RDI 2.0.0 on Amazon Web Services (AWS), with three\navailability zones. It has one pipeline, no sources, and two control-plane\nnodes. Each processor requests one central processing unit (CPU) and 5 GiB\nof memory. Temporary workspace scaling overrides allow the configured counts\nin the table. The allocation window is 10 minutes per scaling request.\n\nConfigured replicas are the requested count. Ready replicas are running\nprocessors that pass readiness checks. Both columns count processors only.\nThey exclude source collectors and control-plane nodes. These resources\nalso use workspace addresses.\n\n| Workspace CIDR | Configured replicas | Ready replicas | Capacity constraint |\n| --- | ---: | ---: | --- |\n| `/22` | 15 | 14 | Address blocks exhausted in the assigned subnet. Another zone retained a block. |\n| `/21` | 50 | 32 | Address blocks exhausted in all worker subnets. Allocation requests were also throttled. |\n| `/20` | 50 | 50 | Capacity for at least 50 replicas in this configuration. The upper limit is not established. |\n\nThe `/22` and `/21` figures use gradual scaling. The `/20` figure uses a\ndirect request for 50 replicas.\n\nThe counts of 14 and 32 are not fixed workspace limits. Address distribution\nacross zones and allocation throttling also affect scaling. The table does\nnot define supported replica limits, throughput, or combined source and\nprocessor capacity. Reserve addresses for collectors, other pipelines,\ngrowth, and maintenance.\n\nYou cannot enlarge an existing workspace CIDR. A larger range requires\n[workspace recreation](https://redis.io/docs/latest/operate/rc/rdi/faq#change-workspace-cidr). See\n[Create a workspace](https://redis.io/docs/latest/operate/rc/rdi/create-workspace) for CIDR\nselection and connectivity requirements."
    },
    {
      "id": "identify-the-bottleneck",
      "title": "Identify the bottleneck",
      "role": "content",
      "text": "Use the **Dashboard** to compare **Throughput**, **Pending**, **Processor load**,\nand **RDI database load** before you change capacity:\n\n- If ingestion from a source is slow and the processor has spare capacity,\n  check the source's diagnostics in **Metrics** before tuning collector\n  properties.\n- If the processor stays at capacity while pending records grow, consider\n  increasing processor replicas.\n- If the processor is backpressured, check target database performance and\n  connectivity before adding processor replicas.\n- If the RDI database stays full or reports out-of-memory (OOM) errors, check\n  ingestion and processing rates, database memory, and throughput. OOM alone\n  does not identify which capacity setting to change. See\n  [Increase RDI database throughput](#increase-rdi-database-throughput).\n\nChange one setting at a time and compare the same representative workload.\nConfirm that the change improves throughput or reduces pending records before\nmaking another increase."
    },
    {
      "id": "increase-collector-ingestion-capacity",
      "title": "Increase collector ingestion capacity",
      "role": "content",
      "text": "Tune collector properties when the collector cannot ingest data from the\nsource quickly enough. These properties are specific to each source. In the\nCloud console, select the source in **Configuration**, then select **Edit\ncollector properties**.\n\nThe console exposes these commonly tuned Debezium properties when they apply to\nthe selected source:\n\n| Collector section | Property | Effect |\n| --- | --- | --- |\n| Source | `snapshot.max.threads` | Increases the threads used for the initial snapshot. |\n| Source | `snapshot.fetch.size` | Increases the rows fetched in each snapshot batch. |\n| Source | `max.batch.size` | Increases the records processed in each batch. |\n| Source | `max.queue.size` | Increases the records buffered in collector memory. |\n| Source | `poll.interval.ms` | Sets the wait before the collector polls for change data capture (CDC) changes. |\n| Source | `record.processing.threads` | Increases the threads that process captured records. |\n| Sink | `redis.batch.size` | Increases the records written to the RDI database in each batch. |\n| Sink | `redis.flush.interval.ms` | Reduces the maximum wait before the collector flushes a batch. |\n\nUse only properties shown for the selected source. The available properties can\nvary by source database.\n\nCloud RDI collectors have 2 CPUs and 8 GB of\nrandom access memory (RAM). You cannot select a different collector size.\nTest each change with a representative workload before you use it in production.\n\nLarger batches and queues use more collector memory. More snapshot and\nrecord-processing threads share the same 2 CPUs. A shorter poll interval can\nreduce CDC latency but can increase load on the source database. Change one\nproperty at a time and use the Dashboard to compare throughput, pending\nrecords, and processor load."
    },
    {
      "id": "increase-rdi-database-throughput",
      "title": "Increase RDI database throughput",
      "role": "content",
      "text": "Each pipeline has an RDI database. When the RDI database limits ingestion or\nprocessing, increase its memory limit or throughput in its **Performance**\nsettings."
    },
    {
      "id": "relevant-metrics",
      "title": "Relevant metrics",
      "role": "content",
      "text": "To view the RDI database metrics, open the RDI database in the Cloud console\nand select the **Metrics** tab. Check these metrics:\n\n- **Used memory**: The RDI database holds records until the processor writes\n  them to the target database. Large rows and high data volumes can fill the\n  RDI database faster than the processor drains it. When the RDI database\n  reaches its memory limit, it rejects writes with OOM errors and the\n  collectors wait.\n- **Ops/sec**: If **Ops/sec** stays close to the RDI database's throughput\n  value, or **Latency** keeps rising, the RDI database cannot handle more\n  writes."
    },
    {
      "id": "single-pipeline",
      "title": "Single pipeline",
      "role": "content",
      "text": "For a single pipeline, check **Used memory** and **Ops/sec**:\n\n- If **Used memory** reaches the memory limit, increase **Memory limit**.\n- If **Ops/sec** stays close to the throughput value, increase **Throughput**."
    },
    {
      "id": "multiple-pipelines",
      "title": "Multiple pipelines",
      "role": "content",
      "text": "For multiple pipelines, check **Used memory** and **Ops/sec** the same way as\nfor a single pipeline.\n\nWhen you increase **Throughput**, choose a value with as many shards as you\nhave pipelines. Each pipeline writes to its own hash slot, and all sources in\na pipeline share that slot. RDI spreads the pipelines' hash slots across the\nslot range, so when the number of shards matches the number of pipelines,\neach pipeline's slot is on a different shard.\n\n| Throughput | Nodes | Shards |\n| --- | ---: | ---: |\n| 25,000 ops/sec (default) | 1 | 1 |\n| 50,000 ops/sec | 2 | 2 |\n| 75,000 ops/sec | 3 | 3 |"
    },
    {
      "id": "edit-the-performance-settings",
      "title": "Edit the performance settings",
      "role": "content",
      "text": "1. In the Cloud console, open the RDI database for the pipeline.\n1. Edit the database **Performance** settings.\n1. Increase **Memory limit** or **Throughput** to the next available value.\n1. Check the RDI database **Metrics** tab and the pipeline Dashboard before\n   you make another increase.\n\nStop increasing the values when the pipeline no longer improves. More RDI\ndatabase throughput does not resolve a bottleneck in another component, so\ncheck processor and target database performance too."
    },
    {
      "id": "increase-processor-capacity",
      "title": "Increase processor capacity",
      "role": "content",
      "text": "1. From the Cloud RDI **Pipelines** list, select the pipeline.\n1. Select the **Settings** tab.\n1. Select **Edit**.\n1. Add or update the `advanced.resources.taskManager.replicas` property with\n   the number of processor replicas you need. Use a positive whole number\n   accepted by the console and check [workspace network capacity](#plan-workspace-network-capacity)\n   before you increase it.\n1. Select **Save changes**.\n1. Select **Apply and restart** to apply the change. This temporarily\n   interrupts processing.\n\nThis property sets the desired processor replica count. After the restart,\ncheck the running count and pipeline performance. More replicas do not\nguarantee higher throughput.\n\nFor application programming interface (API) configuration, the same property\nis represented as follows:\n\n[code example]"
    },
    {
      "id": "confirm-the-applied-capacity",
      "title": "Confirm the applied capacity",
      "role": "content",
      "text": "The **Dashboard** shows the processor replica count next to the processor\nstatus. It shows the running count and configured count in the format\n`running / configured replicas`. For example, `3 / 3 replicas` means three\nTaskManagers are running. The configured count is also three. If only one\ncount is available, the Dashboard labels it as running or configured.\n\nThe running count can be lower while a restart is in progress. If it remains\nbelow the configured count after the restart, check pipeline and source status\nfor errors and contact [Redis support](https://redis.io/support/). Network\ncapacity can prevent additional replicas from starting. Increasing the\nconfigured count again does not resolve a network capacity shortage.\n\nThe console metrics show data-stream record counts and pending records. They do\nnot include a TaskManager replica-count metric.\n\nFor programmatic confirmation, use the [Redis Cloud API](https://redis.io/docs/latest/operate/rc/api/api-reference#tag/Data-Integration)\nto send a `GET` request to\n`/v1/subscriptions/<subscription-id>/data-integration-workspace/rdi/api/v2/pipelines/<pipeline-name>/status`.\nInspect the `flink-processor` entry in `components`. Its `replicas` value is\nthe number of ready processor replicas. For example:\n\n[code example]\n\nBilling is not a real-time way to confirm that a scaling change has completed.\nUse the Dashboard or the status API instead."
    }
  ],
  "examples": [
    {
      "id": "increase-processor-capacity-ex0",
      "language": "yaml",
      "code": "processors:\n  advanced:\n    resources:\n      taskManager:\n        replicas: 3",
      "section_id": "increase-processor-capacity"
    },
    {
      "id": "confirm-the-applied-capacity-ex0",
      "language": "json",
      "code": "{\n  \"name\": \"flink-processor\",\n  \"replicas\": 3\n}",
      "section_id": "confirm-the-applied-capacity"
    }
  ]
}