Recover a Redis Enterprise cluster on Kubernetes

This task describes how to recover a Redis Enterprise cluster on Kubernetes.

When a Redis Enterprise cluster loses contact with more than half of its nodes either because of failed nodes or network split, the cluster stops responding to client connections. When this happens, you must recover the cluster to restore the connections.

You can also perform cluster recovery to reset cluster nodes, to troubleshoot issues, or in a case of active/passive failover.

The Redis Enterprise for Kubernetes automates these recovery steps:

  1. Recreates a fresh Redis Enterprise cluster
  2. Mounts the persistent storage with the recovery files from the original cluster to the nodes of the new cluster
  3. Recovers the cluster configuration on the first node in the new cluster
  4. Joins the remaining nodes to the new cluster.
Warning:
Redis Enterprise for Kubernetes 7.2.4-2 introduces a new limitation. You cannot recover or upgrade your cluster if there are databases with old module versions or manually uploaded modules. See the Redis Enterprise Software 7.2.4 known limitations for more details.

Prerequisites

Recover a cluster

  1. Edit the REC resource to set the clusterRecovery flag to true.

    kubectl patch rec <cluster-name> --type merge --patch '{"spec":{"clusterRecovery":true}}'

  2. Wait for the cluster to recover until it is in the "Running" state.

    To see the state of the cluster, run:

    watch "kubectl describe rec | grep State"

  3. To recover the database, see Recover a failed database.

RATE THIS PAGE
Back to top ↑