{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/architecture",
  "title": "Architecture",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture/",
  "summary": "Discover the main components of RDI",
  "content": "\n## Overview\n\nRDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in a\nnon-Redis *source* database and makes corresponding changes to a Redis\n*target* database. You can use the target as a cache to improve performance\nbecause it will typically handle read queries much faster than the source.\n\nTo use RDI, you define a *dataset* that specifies which data items\nyou want to capture from the source and how you want to\nrepresent them in the target. For example, if the source is a\nrelational database then you specify which table columns you want\nto capture but you don't need to store them in an equivalent table\nstructure in the target. This means you can choose whatever target\nrepresentation is most suitable for your app. To convert from the\nsource to the target representation, RDI applies *transformations*\nto the data after capture.\n\nRDI synchronizes the dataset between the source and target using\na *data pipeline* that implements several processing steps\nin sequence:\n\n1.  A *CDC collector* captures changes to the source database. RDI\n    currently uses an open source collector called\n    [Debezium](https://debezium.io/) for this step.\n\n1.  The collector records the captured changes using\n[Redis streams](https://redis.io/docs/latest/develop/data-types/streams)\n    in the RDI database.\n\n1.  A *stream processor* reads data from the streams and applies\n    any transformations that you have defined (if you don't need\n    any custom transformations then it uses defaults).\n    It then writes the data to the target database for your app to use.\n\nNote that the RDI control processes run on dedicated virtual machines (VMs)\noutside the Redis\nEnterprise cluster where the target database is kept. However, RDI keeps\nits state and configuration data and also the change data streams in a Redis database on the same cluster as the target. The following diagram shows the pipeline steps and the path the data takes on its way from the source to the target:\n\n![images/rdi/ingest/ingest-dataflow.webp](https://redis.io/docs/latest/images/rdi/ingest/ingest-dataflow.webp)\n\nWhen you first start RDI, the target database is empty and so all\nof the data in the source database is essentially \"change\" data.\nRDI collects this data in a phase called *initial cache loading*,\nwhich can take minutes or hours to finish, depending on the size\nof the source data. Once the initial cache loading is complete,\nthere is a *snapshot* dataset in the target that will gradually\nchange when new data gets captured from the source. At this point,\nRDI automatically enters a second phase called *change streaming*, where\nchanges in the data are captured as they happen. Changes are usually\nadded to the target within a few seconds after capture.\n\n## At-least-once delivery guarantee\n\nRDI guarantees *at-least-once delivery* to the target. This means that\na given change will never be lost, but it might be added to the target\nmore than once. Apart from a slight performance overhead, adding a\nchange multiple times is harmless because the multiple writes\nare [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is\nto say that all writes after the first one make no change to the\noverall state).\n\n## Checkpointing\n\nRDI uses Redis streams to store the sequence of change events\ncaptured from the source. The events are then retrieved in order\nfrom the streams, processed, and written to the target. The stream\nprocessor uses a *checkpoint* mechanism to keep track of the last\nevent in the sequence that it has successfully processed and stored. If the processor fails\nfor any reason, it can restart from the last checkpoint and\nre-process any events that might not have been written to the target.\nThis ensures that all changes are eventually recorded, even in the\nface of failures.\n\n## Backpressure mechanism\n\nSometimes, data records can get added to the streams faster than RDI can\nprocess them. This can happen if the target is slowed or disconnected\nor simply if the source quickly generates a lot of change data.\nIf this continues, then the streams will eventually occupy all the\navailable memory. When RDI detects this situation, it applies a\n*backpressure* mechanism to slow or stop the flow of incoming data.\nChange data is held at the source until RDI clears the backlog and has\nenough free memory to resume streaming.\n\n\u003e [!NOTE]\n\u003e The Debezium log sometimes reports that RDI has run out\n\u003e of memory (usually while creating the initial snapshot). This is not\n\u003e an error, just an informative message to note that RDI has applied\n\u003e the backpressure mechanism.\n\n## Supported sources\n\nRDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors:\n\n| Database | Versions | AWS RDS  Versions | GCP SQL Versions |\n| :-- | :-- | :-- | :-- |\n| Oracle | 19c, 21c, 23ai (LogMiner only) | 19c, 21c | - |\n| MariaDB | 10.5, 11.4.x, 11.7.x | 10.4 to 10.11 | - |\n| MongoDB | 6.0, 7.0, 8.0 | - | - |\n| MySQL | 5.7, 8.0.x, 8.4.x, 9.0, 9.1 | 8.0.x | 8.0 |\n| PostgreSQL | 10, 11, 12, 13, 14, 15, 16, 17, 18 | 11, 12, 13, 14, 15, 16, 17, 18 | 15 |\n| Supabase (uses PostgreSQL) | 10, 11, 12, 13, 14, 15, 16, 17  | - | - |\n| SQL Server | 2017, 2019, 2022 | 2016, 2017, 2019, 2022 | 2019 |\n| Spanner | - | - | All versions |\n| AlloyDB for PostgreSQL | 14.2, 15.7 | - | 14.2, 15.7 |\n| AWS Aurora/PostgreSQL | 15 | 15 | - |\n| Neon | 14, 15, 16, 17 | - | - |\n| Snowflake (preview) | - | - | - |\n\n\n## How RDI is deployed\n\nRDI is designed with three *planes* that provide its services.\n\nThe *control plane* contains the processes that keep RDI active.\nIt includes:\n\n-   An *API server* process that exposes a REST API to observe and control RDI.\n-   An *operator* process that manages the *data plane* processes.\n-   A *metrics exporter* process that reads metrics from the RDI database\n    and exports them as [Prometheus](https://prometheus.io/) metrics.\n\nThe *data plane* contains the processes that actually move the data.\nIt includes the *CDC collector* and the *stream processor* that implement\nthe two phases of the pipeline lifecycle (initial cache loading and change streaming).\n\nThe *management plane* provides tools that let you interact\nwith the control plane.\n\n-   Use the CLI tool to install and administer RDI and to deploy\n    and manage a pipeline.\n-   Use the pipeline editor included in Redis Insight to design\n    or edit a pipeline.\n\nThe diagram below shows all RDI components and the interactions between them:\n\n![images/rdi/ingest/ingest-control-plane.webp](https://redis.io/docs/latest/images/rdi/ingest/ingest-control-plane.webp)\n\n## Stream processor implementations\n\nRDI provides two implementations of the stream processor, *classic* and\n*Flink*. You select the implementation per pipeline through the\n[`processors.type`](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors)\nproperty in `config.yaml`. The default is `classic`, so existing pipelines\nkeep their behavior unchanged.\n\nSee\n[Differences between the classic and Flink processors](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink)\nfor a side-by-side comparison and\n[Migrate from the classic processor to the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink)\nfor guidance on migrating an existing pipeline to the Flink processor.\n\n## VM and Kubernetes deployments\n\nThe following sections describe the VM configurations you can use to\ndeploy RDI.\n\n### RDI on your own VMs\n\nFor this deployment, you must provide two VMs. The collector and stream processor\nare active on one VM, while on the other they are in standby to provide high availability.\nThe two operators running on both VMs use a leader election algorithm to decide which\nVM is the active one (the \"leader\").\nThe diagram below shows this configuration:\n\n![images/rdi/ingest/ingest-active-passive-vms.webp](https://redis.io/docs/latest/images/rdi/ingest/ingest-active-passive-vms.webp)\n\nSee [Install on VMs](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/install-vm)\nfor more information.\n\n### RDI on Kubernetes\n\nYou can use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) to install\non [Kubernetes (K8s)](https://kubernetes.io/), including Red Hat\n[OpenShift](https://docs.openshift.com/). This creates:\n\n-   A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`.\n    You can also use a different namespace name if you prefer.\n-   [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and\n    [services](https://kubernetes.io/docs/concepts/services-networking/service/) for the\n    [RDI operator](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture#how-rdi-is-deployed),\n    [metrics exporter](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/observability), and API server.\n-   A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/)\n    and [RBAC resources](https://kubernetes.io/docs/reference/access-authn-authz/rbac) for the RDI operator.\n-   A [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) with RDI database details.\n-   [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)\n    with the RDI database credentials and TLS certificates.\n-   Other optional K8s resources such as [ingresses](https://kubernetes.io/docs/concepts/services-networking/ingress/)\n    that can be enabled depending on your K8s environment and needs.\n\nSee [Install on Kubernetes](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/install-k8s)\nfor more information.\n\n### Secrets and security considerations\n\nThe credentials for the database connections, as well as the certificates\nfor [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) and\n[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) are saved in K8s secrets.\nRDI stores all state and configuration data inside the Redis Enterprise cluster\nand does not store any other data on your RDI VMs or anywhere else outside the cluster.\n",
  "tags": ["docs","integrate","rs","rdi"],
  "last_updated": "2026-09-19T17:55:58-07:00",
  "children": [{"id":"integrate/redis-data-integration/architecture/classic-vs-flink","summary":"Compare the classic and Flink stream processor implementations.","title":"Differences between the classic and Flink processors","url":"https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink/"}]
}