Observability

Learn how to monitor RDI

RDI reports metrics about its operation using Prometheus exporter endpoints. You can connect to the endpoints with Prometheus to query the metrics and plot simple graphs or with Grafana to produce more complex visualizations and dashboards.

RDI exposes three endpoints:

• Collector metrics: CDC collector performance and connectivity
• Stream processor metrics: Data processing performance and throughput
• Operator metrics: Kubernetes operator health and Pipeline resource states

The sections below explain these sets of metrics in more detail. See the architecture overview for an introduction to these concepts.

Accessing the metrics

The way you access the metrics endpoints depends on whether you are using a VM installation or a Helm installation for RDI. The sections below describe the correct approach for each installation type.

VM Installation

For VM installations, the metrics are available by default on the following endpoints:

• Collector metrics: https://<RDI_HOST>/collector-source/metrics
• Stream processor metrics: https://<RDI_HOST>/metrics
• Operator metrics: https://<RDI_HOST>/operator/metrics

Please note that for RDI versions prior to 1.16.0 the collector metrics are not accessible.

Helm installation

For Helm installations, the metrics are available via autodiscovery in the K8s cluster. Follow the steps below to use them:

1. Make sure you have the Prometheus Operator installed in your K8s cluster (see the Prometheus Operator installation guide for more information about this).

2. Update your values.yaml file to enable metrics for the operator, collector and stream processor components.

   • For the collector, update the collector section, under the dataPlane section:

     dataPlane:
       collector:
         # Enable service monitor
         serviceMonitor:
           enabled: true
     
           # Make sure to label the ServiceMonitor so that Prometheus can discover it
           labels:
             release: prometheus
     

   • For the stream processor, update the rdiMetricsExporter section:

     rdiMetricsExporter:
       # Enable service monitor
       serviceMonitor:
         enabled: true
     
         # Make sure to label the ServiceMonitor so that Prometheus can discover it
         labels:
           release: prometheus
     

   • For the operator, update the operator section:

     operator:
       prometheus:
         enabled: true
         labels:
           release: prometheus
       metrics:
         enabled: true
     

Collector metrics

These metrics are divided into three groups:

• Pipeline state: metrics about the pipeline mode and connectivity
• Data flow counters: counters for data breakdown per source table
• Processing performance: processing speed of RDI micro batches

The following table lists all collector metrics and their descriptions:

Stream processor metrics

RDI reports metrics during the two main phases of the ingest pipeline, the snapshot phase and the change data capture (CDC) phase. (See the pipeline lifecycle docs for more information). The table below shows the full set of metrics that RDI reports with their descriptions.

Operator metrics

The RDI operator exposes Prometheus metrics at the /metrics endpoint to monitor the health and state of the operator itself and the Pipeline resources it manages.

The endpoint for operator metrics is https://<RDI_HOST>/operator/metrics (or the operator service endpoint in Kubernetes environments).

Operator metric types

Most of the metrics exposed by the RDI operator are standard controller-runtime metrics. The metrics that are relevant for RDI operations are listed in the table below:

Understanding operator metrics

Pipeline phase tracking: The rdi_operator_pipeline_phase metric helps you monitor the lifecycle state of each RDI Pipeline resource. Each pipeline reports its current phase (Active, Inactive, Pending, Resetting, or Error) as a gauge value of 1, while all other phases for that pipeline are set to 0. This allows you to track phase transitions and identify pipelines that are stuck in error states.

Leader election: In high availability (HA) or disaster recovery (DR) deployments with multiple RDI instances, the rdi_operator_is_leader metric indicates which RDI instance is actively managing Pipeline resources. Only one RDI instance should have a value of 1 at any time, while all other instances should report 0. This metric is useful for troubleshooting leader election issues in HA or DR deployments.

Accessing operator metrics

In Kubernetes deployments, you can configure Prometheus to scrape operator metrics by enabling the Prometheus ServiceMonitor in your Helm values:

operator:
  prometheus:
    enabled: true
    labels:
      release: prometheus

Note: The ServiceMonitor resources must be labelled correctly for metrics to be auto-scraped by Prometheus. The correct label is configured in Prometheus, by default it is release: prometheus. You can also expose the metrics endpoint externally using an Ingress:

operator:
  ingress:
    enabled: true
    hosts:
      - operator.example.com
    pathPrefix: ""

Then access metrics at https://operator.example.com/operator/metrics.

Recommended alerting strategy

The alerting strategy described in the sections below focuses on system failures and data integrity issues that require immediate attention. Most other metrics are informational, so you should monitor them for trends rather than trigger alerts.

Critical alerts (immediate response required)

These are the only alerts that require immediate action:

Collector alerts:

• Connected = 0: Database connectivity has been lost. RDI cannot function without a database connection.
• NumberOfErroneousEvents > 0: Errors are occurring during data processing. This indicates data corruption or processing failures.
• SnapshotAborted = 1: The snapshot process has failed, so the initial sync is incomplete.

Processor alerts:

• rejected_records_total > 0: Records are being rejected. This indicates data quality issues or processing failures.
• rdi_engine_state: Alert only if the state indicates a clear failure condition (not just "not running").

Operator alerts:

• rdi_operator_pipeline_phase with phase="Error" for more than 2 minutes: A Pipeline resource has entered an error state and requires investigation.
• No leader in HA or DR setups: If both RDI instances report rdi_operator_is_leader = 0 for more than 2 minutes, the RDI pipeline is not active.
• Multiple leaders in HA or DR setups: If both RDI instances report rdi_operator_is_leader = 1, RDI is in a "split brain" state.

Important monitoring (but not alerts)

You should monitor these metrics on dashboards and review them regularly, but they don't require automated alerts:

• Queue metrics: Queue utilization can vary widely and hitting 0% or 100% capacity may be normal during certain operations.
• Latency metrics: Lag and processing times depend heavily on business requirements and normal operational patterns.
• Event counters: Event rates naturally vary based on application usage patterns.
• Snapshot progress: Snapshot duration and progress depend on data size, so you should typically monitor them manually.
• Schema changes: Schema change frequency is highly application-dependent.

Key principles for RDI alerting

• Alert on failures, not performance: Focus alerts on system failures rather than performance degradation.
• Business context matters: Latency and throughput requirements vary significantly between organizations.
• Establish baselines first: Monitor metrics for weeks before you set any threshold-based alerts.
• Avoid alert fatigue: If you see too many non-critical alerts, you are less likely to take truly critical issues seriously.
• Use dashboards for trends: Most metrics are better suited for dashboard monitoring than alerting

Monitoring best practices

• Dashboard-first approach: Use Grafana dashboards to visualize trends and patterns.
• Baseline establishment: Monitor your specific workload for 2-4 weeks before you consider adding more alerts.
• Business SLA alignment: Only create alerts for metrics that directly impact your business SLA requirements.
• Manual review: Don't use automated alerts to review metric trends. Instead, schedule regular business reviews to check them manually.

RDI logs

RDI uses fluentd and logrotate to ship and rotate logs for its Kubernetes (K8s) components. So whenever a containerized component is removed by the RDI operator process or by K8s, the logs are available for you to inspect. By default, RDI stores logs in the host VM file system at /opt/rdi/logs. The logs are recorded at the minimum INFO level and get rotated when they reach a size of 100MB. RDI retains the last five log rotated files by default. Logs are in a straightforward text format, which lets you analyze them with several different observability tools. You can change the default log settings using the redis-di configure-rdi command.

Dump support package

If you ever need to send a comprehensive set of forensics data to Redis support then you should run the redis-di dump-support-package command from the CLI. See Troubleshooting for more information.