{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/installation/migration-classic-to-flink",
  "title": "Migrate from the classic processor to the Flink processor",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink/",
  "summary": "Learn how to migrate an existing RDI pipeline from the classic processor to the Apache Flink-based processor.",
  "content": "\nRDI ships with two stream processor implementations. The default *classic*\nprocessor is implemented in Python. The *Flink* processor is built on top of\n[Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes\ninstallations. The Flink processor can achieve much higher throughput\nduring snapshots, scales horizontally by changing the number of TaskManager replicas,\nand uses Flink checkpointing for fault tolerance. See [Stream processor implementations](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations)\nfor an overview.\n\nThis page describes how to migrate an existing pipeline from the classic\nprocessor to the Flink processor. The steps are the same on VMs and Kubernetes,\nexcept for the optional Helm-level tuning in [Step 1](#step-1-configure-the-flink-processor-at-the-helm-chart-level-kubernetes),\nwhich applies to Kubernetes only.\n\n## Before you migrate\n\nConfirm that your pipeline is compatible with the Flink processor:\n\n-   `JSON.MERGE` semantics differ from the classic processor's Lua-based merge\n    when null values are involved (see\n    [`use_native_json_merge`](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors)).\n    The Flink processor always uses the native `JSON.MERGE` command when the\n    target database supports it.\n-   Ensure your Kubernetes cluster or VM has enough capacity for the Flink JobManager\n    and TaskManager pods (see\n    [Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor)\n    for the default sizing).\n\n## Step 1: Configure the Flink processor at the Helm chart level (Kubernetes)\n\nThis step applies to **Kubernetes** installations only. On VM installations,\nskip it and enable the Flink processor per pipeline in step 2.\n\nThe Flink processor is always available — no opt-in is required at the Helm\nchart level. The defaults are sized for typical workloads, so you can skip\nthis step if you don't need to override them. To adjust the JobManager and\nTaskManager defaults, add an `operator.dataPlane.flinkProcessor` block to\nyour `rdi-values.yaml` file and run `helm upgrade` as described in\n[Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor).\nExisting pipelines continue to run on the classic processor until you switch\nthem in step 2.\n\nFor VM installations, skip this step. You can configure per-pipeline Flink\nresources in step 4.\n\n## Step 2: Switch the pipeline to the Flink processor\n\nIn the pipeline's `config.yaml`, set\n[`processors.type`](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors)\nto `flink`:\n\n```yaml\nprocessors:\n  type: flink\n  ...\n```\n\nThen redeploy the pipeline. The operator stops the classic processor pods\nand starts the Flink JobManager and TaskManager workloads for the pipeline.\n\n## Step 3: Adapt deprecated and classic-only properties\n\nSome `processors` properties are no-ops, classic-only, or have moved to\n`processors.advanced` for the Flink processor. The following table lists the\nproperties that need attention when migrating.\n\n| Property | Action when migrating to Flink |\n| :-- | :-- |\n| `on_failed_retry_interval` | No-op. Remove. |\n| `duration` | No-op. Use `read_batch_timeout_ms` instead. |\n| `dedup`, `dedup_max_size`, `dedup_strategy` | Classic-only. Remove. |\n| `enable_async_processing`, `batch_queue_size`, `ack_queue_size` | Classic-only. Remove. |\n| `initial_sync_processes` | Classic-only. Configure parallelism through `advanced.flink.taskmanager.numberOfTaskSlots` and `advanced.resources.taskManager.replicas` instead. |\n| `idle_streams_check_interval_ms`, `busy_streams_check_interval_ms` | Classic-only. Use `processors.advanced.source.discovery.interval.ms` for a single discovery interval. |\n| `idle_sleep_time_ms` | Classic-only. Remove. |\n| `use_native_json_merge` | Classic-only. The Flink processor always uses `JSON.MERGE` when the target supports it. |\n\nThe classic processor silently ignores `processors.advanced`,\nand the Flink processor silently ignores classic-only top-level properties, so keeping\nboth top-level properties and their `processors.advanced` equivalents lets\nyou switch back without further edits.\n\n## Step 4: Tune the Flink processor (optional)\n\nFine-tune the Flink processor through the `processors.advanced` section.\nFor example:\n\n```yaml\nprocessors:\n  type: flink\n  advanced:\n    source:\n      # Time between checks for new input streams.\n      discovery.interval.ms: 1000\n    flink:\n      # Number of parallel task slots per TaskManager pod.\n      taskmanager.numberOfTaskSlots: 2\n      # Total memory budget for each TaskManager JVM process.\n      taskmanager.memory.process.size: 4096m\n    resources:\n      taskManager:\n        # Number of TaskManager pods.\n        replicas: 2\n```\n\nSee the\n[`processors.advanced` reference](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors)\nfor the full set of available properties.\n\n## Step 5: Update observability\n\nThe Flink processor exposes Prometheus metrics directly\nfrom the Flink JobManager and TaskManager pods.\nSee\n[Flink processor metrics](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/observability#flink-processor-metrics)\nfor the `ServiceMonitor` configuration and the available metrics.\n\n## Rolling back\n\nTo revert a pipeline to the classic processor, set `processors.type` back to\n`classic` (or remove the property) and redeploy the pipeline. The\n`processors.advanced` section is silently ignored by the classic processor,\nso you don't need to remove it before switching back.\n",
  "tags": ["docs","integrate","rs","rdi"],
  "last_updated": "2026-09-19T17:55:58-07:00"
}