Observability
Learn how to monitor RDI ingest
RDI reports metrics about its operation using Prometheus exporter endpoints. You can connect to the endpoints with Prometheus to query the metrics and plot simple graphs or with Grafana to produce more complex visualizations and dashboards.
RDI exposes two endpoints, one for CDC collector metrics and another for stream processor metrics. The sections below explain these sets of metrics in more detail. See the architecture overview for an introduction to these concepts.
redis-di status
command from the CLI.Collector metrics
The endpoint for the collector metrics is https://<RDI_HOST>/metrics/collector-source
These metrics are divided into three groups:
- Pipeline state: metrics about the pipeline mode and connectivity
- Data flow counters: counters for data breakdown per source table
- Processing performance: processing speed of RDI micro batches
Stream processor metrics
The endpoint for the stream processor metrics is https://<RDI_HOST>/metrics/rdi
RDI reports metrics during the two main phases of the ingest pipeline, the snapshot phase and the change data capture (CDC) phase. (See the ingest pipeline lifecycle docs for more information). The table below shows the full set of metrics that RDI reports.
| Metric | Phase |
|---|---|
| CapturedTables | Both |
| Connected | CDC |
| LastEvent | Both |
| LastTransactionId | CDC |
| MilliSecondsBehindSource | CDC |
| MilliSecondsSinceLastEvent | Both |
| NumberOfCommittedTransactions | CDC |
| NumberOfEventsFiltered | Both |
| QueueRemainingCapacity | Both |
| QueueTotalCapacity | Both |
| RemainingTableCount | Snapshot |
| RowsScanned | Snapshot |
| SnapshotAborted | Snapshot |
| SnapshotCompleted | Snapshot |
| SnapshotDurationInSeconds | Snapshot |
| SnapshotPaused | Snapshot |
| SnapshotPausedDurationInSeconds | Snapshot |
| SnapshotRunning | Snapshot |
| SourceEventPosition | CDC |
| TotalNumberOfCreateEventsSeen | CDC |
| TotalNumberOfDeleteEventsSeen | CDC |
| TotalNumberOfEventsSeen | Both |
| TotalNumberOfUpdateEventsSeen | CDC |
| TotalTableCount | Snapshot |
RDI logs
RDI uses fluentd and
logrotate to ship and rotate logs
for its Kubernetes (K8s) components.
So whenever a containerized component is removed by the RDI operator process or by K8s,
the logs are available for you to inspect.
By default, RDI stores logs in the host VM file system at /opt/rdi/logs.
The logs are recorded at the minimum INFO level and get rotated when they reach a size of 100MB.
RDI retains the last five log rotated files by default.
Logs are in JSON format, which lets you analyze them with several different observability tools.
You can change the default log settings using the
redis-di config-rdi
command.
Support package
If you ever need to send a comprehensive set of forensics data to Redis support then you should
run the
redis-di dump-support-package
command from the CLI.
This command gathers the following data:
- All the internal RDI components and their status
- All internal RDI configuration
- List of secret names used by RDI components (but not the secrets themselves)
- RDI logs
- RDI component versions
- Output from the
redis-di statuscommand - Text of the
config.yamlfile - Text of the Job configuration files
- [optional] RDI DLQ streams content
- Rejected records along with the reason for their rejection (should not exist in production)