{
  "id": "migration-classic-to-flink",
  "title": "Migrate from the classic processor to the Flink processor",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/installation/migration-classic-to-flink/",
  "summary": "Learn how to migrate an existing RDI pipeline from the classic processor to the Apache Flink-based processor.",
  "tags": [
    "docs",
    "integrate",
    "rs",
    "rdi"
  ],
  "last_updated": "2026-05-12T09:07:59-04:00",
  "page_type": "content",
  "content_hash": "1fe7cbf0b94b53e3a8a73b73676301cb00b59617b385160cab301a728f196b24",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "RDI ships with two stream processor implementations. The default *classic*\nprocessor is implemented in Python and runs on both VMs and Kubernetes. The\n*Flink* processor is built on top of [Apache Flink](https://flink.apache.org/)\nand currently runs on Kubernetes only. It can achieve much higher throughput\nduring snapshots, scales horizontally by changing the number of TaskManager replicas,\nand uses Flink checkpointing for fault tolerance. See [Stream processor implementations](https://redis.io/docs/latest/integrate/redis-data-integration/architecture#stream-processor-implementations)\nfor an overview.\n\nThis page describes how to migrate an existing pipeline from the classic\nprocessor to the Flink processor.\n\nThe Flink processor is currently supported on Kubernetes only. VM\ninstallations must continue to use the classic processor."
    },
    {
      "id": "before-you-migrate",
      "title": "Before you migrate",
      "role": "content",
      "text": "Confirm that your pipeline is compatible with the Flink processor:\n\n-   The Flink processor supports `hash` and `json` target data types only. If\n    any of your jobs use the `set`, `sorted_set`, `stream`, or `string` data\n    types, those jobs must be rewritten or kept on the classic processor.\n-   `JSON.MERGE` semantics differ from the classic processor's Lua-based merge\n    when null values are involved (see\n    [`use_native_json_merge`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processors)).\n    The Flink processor always uses the native `JSON.MERGE` command when the\n    target database supports it.\n-   Ensure your Kubernetes cluster has enough capacity for the Flink JobManager\n    and TaskManager pods (see\n    [Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/installation/install-k8s#configure-the-flink-processor)\n    for the default sizing)."
    },
    {
      "id": "step-1-configure-the-flink-processor-at-the-helm-chart-level",
      "title": "Step 1: Configure the Flink processor at the Helm chart level",
      "role": "content",
      "text": "The Flink processor is always available — no opt-in is required at the Helm\nchart level. The defaults are sized for typical workloads, so you can skip\nthis step if you don't need to override them. To adjust the JobManager and\nTaskManager defaults, add an `operator.dataPlane.flinkProcessor` block to\nyour `rdi-values.yaml` file and run `helm upgrade` as described in\n[Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/installation/install-k8s#configure-the-flink-processor).\nExisting pipelines continue to run on the classic processor until you switch\nthem in step 2."
    },
    {
      "id": "step-2-switch-the-pipeline-to-the-flink-processor",
      "title": "Step 2: Switch the pipeline to the Flink processor",
      "role": "content",
      "text": "In the pipeline's `config.yaml`, set\n[`processors.type`](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#processors)\nto `flink`:\n\n[code example]\n\nThen redeploy the pipeline. The operator stops the classic processor pods\nand starts the Flink JobManager and TaskManager workloads for the pipeline."
    },
    {
      "id": "step-3-adapt-deprecated-and-classic-only-properties",
      "title": "Step 3: Adapt deprecated and classic-only properties",
      "role": "content",
      "text": "Some `processors` properties are no-ops, classic-only, or have moved to\n`processors.advanced` for the Flink processor. The following table lists the\nproperties that need attention when migrating.\n\n| Property | Action when migrating to Flink |\n| :-- | :-- |\n| `on_failed_retry_interval` | No-op. Remove. |\n| `duration` | No-op. Use `read_batch_timeout_ms` instead. |\n| `dedup`, `dedup_max_size`, `dedup_strategy` | Classic-only. Remove. |\n| `enable_async_processing`, `batch_queue_size`, `ack_queue_size` | Classic-only. Remove. |\n| `initial_sync_processes` | Classic-only. Configure parallelism through `advanced.flink.taskmanager.numberOfTaskSlots` and `advanced.resources.taskManager.replicas` instead. |\n| `idle_streams_check_interval_ms`, `busy_streams_check_interval_ms` | Classic-only. Use `processors.advanced.source.discovery.interval.ms` for a single discovery interval. |\n| `idle_sleep_time_ms` | Classic-only. Remove. |\n| `use_native_json_merge` | Classic-only. The Flink processor always uses `JSON.MERGE` when the target supports it. |\n\nThe classic processor silently ignores `processors.advanced`,\nand the Flink processor silently ignores classic-only top-level properties, so keeping\nboth top-level properties and their `processors.advanced` equivalents lets\nyou switch back without further edits."
    },
    {
      "id": "step-4-tune-the-flink-processor-optional",
      "title": "Step 4: Tune the Flink processor (optional)",
      "role": "content",
      "text": "Fine-tune the Flink processor through the `processors.advanced` section.\nFor example:\n\n[code example]\n\nSee the\n[`processors.advanced` reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processors)\nfor the full set of available properties."
    },
    {
      "id": "step-5-update-observability",
      "title": "Step 5: Update observability",
      "role": "content",
      "text": "The Flink processor exposes Prometheus metrics directly\nfrom the Flink JobManager and TaskManager pods.\nSee\n[Flink processor metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#flink-processor-metrics)\nfor the `ServiceMonitor` configuration and the available metrics."
    },
    {
      "id": "rolling-back",
      "title": "Rolling back",
      "role": "content",
      "text": "To revert a pipeline to the classic processor, set `processors.type` back to\n`classic` (or remove the property) and redeploy the pipeline. The\n`processors.advanced` section is silently ignored by the classic processor,\nso you don't need to remove it before switching back."
    }
  ],
  "examples": [
    {
      "id": "step-2-switch-the-pipeline-to-the-flink-processor-ex0",
      "language": "yaml",
      "code": "processors:\n  type: flink\n  ...",
      "section_id": "step-2-switch-the-pipeline-to-the-flink-processor"
    },
    {
      "id": "step-4-tune-the-flink-processor-optional-ex0",
      "language": "yaml",
      "code": "processors:\n  type: flink\n  advanced:\n    source:\n      # Time between checks for new input streams.\n      discovery.interval.ms: 1000\n    target:\n      # Verify writes are replicated before acknowledging.\n      wait.enabled: true\n      wait.write.timeout.ms: 1000\n    flink:\n      # Number of parallel task slots per TaskManager pod.\n      taskmanager.numberOfTaskSlots: 2\n      # Total memory budget for each TaskManager JVM process.\n      taskmanager.memory.process.size: 4096m\n    resources:\n      taskManager:\n        # Number of TaskManager pods.\n        replicas: 2",
      "section_id": "step-4-tune-the-flink-processor-optional"
    }
  ]
}