Client-side geographic failover

Improve reliability using the failover features of StackExchange.Redis.

StackExchange.Redis supports Client-side geographic failover to improve the availability of connections to Redis databases. This page explains how to configure StackExchange.Redis for failover. For an overview of the concepts, see the main Client-side geographic failover page.

Failover configuration

Failover support is available in StackExchange.Redis v3.1.0 and later. The failover types live in the StackExchange.Redis.Availability namespace, so you should add the following using directives to your source file:

using StackExchange.Redis;
using StackExchange.Redis.Availability;

<NoWarn>$(NoWarn);SER007</NoWarn>

#pragma warning disable SER007

The example below shows a simple case with a list of two servers, redis-east and redis-west, where redis-east is the preferred target. If redis-east fails, StackExchange.Redis should fail over to redis-west.

A failover-capable connection is a group of endpoints, each of which could serve your workload. Create the group using ConnectionMultiplexer.ConnectGroupAsync(), passing an array of ConnectionGroupMember instances instead of a single connection string, as shown in the example below:

// Define your Redis endpoints, with the highest weight being tried first.
ConnectionGroupMember[] members = [
    new("redis-east.example.com:6379", name: "US East") { Weight = 1.0 },
    new("redis-west.example.com:6379", name: "US West") { Weight = 0.5 }
];

// Connect to all members.
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// Use the connection exactly as you would a normal multiplexer.
IDatabase db = conn.GetDatabase();
await db.StringSetAsync("mykey", "myvalue");
RedisValue value = await db.StringGetAsync("mykey");

ConnectGroupAsync() returns a connection that you use just like a standard multiplexer, but it also handles the connection management and failover transparently. The IDatabase and ISubscriber instances obtained from the multiplexer also work with a group as they do with an individual endpoint.

Endpoint configuration

Each endpoint is represented by a ConnectionGroupMember, which you can create from a connection string or from a ConfigurationOptions instance. Use ConfigurationOptions when you need to provide credentials, TLS settings, or other options for each endpoint individually:

var eastConfig = new ConfigurationOptions
{
    EndPoints = { "redis-east-1.example.com:6379", "redis-east-2.example.com:6379" },
    Password = "east-password",
    Ssl = true,
};

var westConfig = new ConfigurationOptions
{
    EndPoints = { "redis-west-1.example.com:6379", "redis-west-2.example.com:6379" },
    Password = "west-password",
    Ssl = true,
};

ConnectionGroupMember[] members = [
    new(eastConfig, name: "US East"),
    new(westConfig, name: "US West")
];

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

The ConnectionGroupMember class provides the following properties to configure each endpoint:

Note that you can also adjust weights at runtime, for example in response to changing conditions:

members[0].Weight = 1;   // Reduce preference for the local DC
members[1].Weight = 10;  // Increase preference for the remote DC

Group configuration

Health checks, circuit breakers, and retries are each configured using immutable policy types (HealthCheck, CircuitBreaker, and RetryPolicy). Each of the three policy types follows the same basic pattern:

1. The policy type is immutable and safe to share between members.
2. Each policy has a nested Builder that carries the mutable settings. A new Builder starts from the default values, so you only set what you want to change. A Builder converts implicitly to its policy type, so you can assign or pass it inline. Alternatively, you can call the Create() method explicitly to create the policy. Create() also validates the values, throwing ArgumentOutOfRangeException or ArgumentException at the point of configuration if a value is invalid.
3. MultiGroupOptions (itself immutable, with its own Builder) holds the group-wide defaults. ConnectionGroupMember contains the same properties as MultiGroupOptions but allows you to override these shared values for each member.

Supply the group-wide defaults by passing a MultiGroupOptions instance as the second argument to ConnectGroupAsync():

HealthCheck healthCheck = new HealthCheck.Builder { ProbeCount = 5 };
CircuitBreaker breaker = new CircuitBreaker.Builder { FailureRateThreshold = 25 };
RetryPolicy retry = new RetryPolicy.Builder { MaxAttempts = 5 };

MultiGroupOptions options = new MultiGroupOptions.Builder
{
    HealthCheck = healthCheck,
    CircuitBreaker = breaker,
    RetryPolicy = retry,
    HealthCheckInterval = TimeSpan.FromSeconds(2),
};

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

Circuit breaker configuration

A circuit breaker passively monitors the traffic already flowing over a connection and closes the connection when it detects that the connection has become unstable (see Detecting connection problems for more information on how the circuit breaker works). Configure the circuit breaker for the whole group using the CircuitBreaker option of MultiGroupOptions:

MultiGroupOptions options = new MultiGroupOptions.Builder
{
    CircuitBreaker = new CircuitBreaker.Builder
    {
        FailureRateThreshold = 25,                   // Trip above 25% failures.
        MinimumNumberOfFailures = 100,               // ...but only after 100 failures in the window.
        MetricsWindowSize = TimeSpan.FromSeconds(5), // Rolling window to measure over.
    }
};

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

The CircuitBreaker.Builder class provides the following properties:

The circuit breaker counts transient and connection-level errors (including timeouts) as failures, but counts application-level errors (such as a WRONGTYPE for a bad command) as successes because they don't indicate an unhealthy connection.

You can also configure a circuit breaker for a single connection (outside a group) using the ConfigurationOptions.CircuitBreaker option. Use CircuitBreaker.None to disable the circuit breaker.

Retry configuration

You can configure an IDatabaseAsync instance to retry commands that fail due to transient errors (such as temporary network delays) using the WithRetry() method:

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// Wrap the database once, then reuse the wrapper like any other IDatabaseAsync.
// The parameterless overload uses the policy configured on the connection.
IDatabaseAsync db = conn.GetDatabase().WithRetry();

// A transient fault is retried automatically. If the group fails over in the
// meantime, the retry lands on the new active member.
RedisValue value = await db.StringGetAsync("mykey");

Note that you can enable retries only for IDatabaseAsync, not for the synchronous equivalent IDatabase.

If you call WithRetry() without parameters, it will use the default retry policy (see Group configuration for details). However, you can also pass a policy explicitly to override the defaults for an individual database:

RetryPolicy policy = new RetryPolicy.Builder
{
    MaxAttempts = 5,
    RetryDelay = TimeSpan.FromMilliseconds(200),
    JitterMax = TimeSpan.FromMilliseconds(100),
};
IDatabaseAsync db = conn.GetDatabase().WithRetry(policy);

The RetryPolicy.Builder class provides the following properties:

The retry mechanism classifies errors in the same way as the circuit breaker (see Circuit breaker configuration). Only transient errors are retried.

Which operations are safe to retry?

Commands are generally safe to retry if they are idempotent (that is to say, multiple invocations of the same command have the same result as a single invocation). This excludes commands like INCR that modify whatever value is currently stored in the database.

Each command belongs to a retry category that describes how "dangerous" it is from the perspective of retrying. Categories are ordered from least to most dangerous, and each has its own CommandFlags value as described in the table below:

A policy only retries commands at or below its MaxCommandRetryCategory level. For the built-in typed methods (such as StringGet, StringSet, and HashSet) StackExchange.Redis assigns the appropriate category automatically.

Arbitrary commands issued using Execute()/ExecuteAsync(), and Lua scripts run using ScriptEvaluate()/ScriptEvaluateAsync(), have side-effects that the library cannot infer, so they are not retried by default. However, you can pass your own choice of retry category using the flags parameter:

// a Lua script that only reads: opt into retries
var value = await db.ScriptEvaluateAsync(
    "return redis.call('GET', KEYS[1])",
    keys: [key],
    flags: CommandFlags.CommandRetryReadOnly
);

Failback configuration

Each member tracks two independent pieces of state:

• IsConnected: the last observed connectivity status of the underlying connection.
• IsUnhealthy: whether the member has been disabled by a failed health check or a tripped circuit breaker.

A member can be selected as active only when it is both connected and healthy.

MultiGroupOptions.FailbackDelay is the minimum interval over which a member must remain healthy (measured from its most recent failure) before it is automatically returned to rotation. This helps to protect against flapping, where a member is only intermittently available between repeated failures.

MultiGroupOptions options = new MultiGroupOptions.Builder
{
    FailbackDelay = TimeSpan.FromMinutes(2), // Must be healthy for 2 minutes after its last failure.
};

FailbackDelay defaults to a value of TimeSpan.Zero, which means the member comes back into rotation as soon as it passes a health check. If you use a value of TimeSpan.MaxValue then automatic failback is effectively disabled. However, you can use the ResetIsUnhealthy() or TryFailoverTo() methods to enable a member manually (see Manual failover for more information).

Health check configuration

Each health check consists of one or more separate probes, each of which is a simple test (such as a PING command) to determine if the member is available. The results of the separate probes are combined using a configurable policy to determine whether the member is healthy. When a member fails its health check, it is flagged as unhealthy and traffic is routed to other healthy members instead. When an unhealthy member recovers, traffic can be routed to it again, after the failback delay has elapsed.

Configure health checks for the whole group using the HealthCheck option of MultiGroupOptions:

HealthCheck healthCheck = new HealthCheck.Builder
{
    ProbeCount = 3,                                 // Maximum probe attempts per check.
    ProbeTimeout = TimeSpan.FromSeconds(3),         // Timeout for each probe attempt.
    ProbeInterval = TimeSpan.FromMilliseconds(500), // Delay between failed probes.
    Probe = HealthCheckProbe.Ping,                  // Which probe type to use.
    ProbePolicy = HealthCheckProbePolicy.AllSuccess // How to evaluate the probe results.
};

MultiGroupOptions options = new MultiGroupOptions.Builder
{
    HealthCheck = healthCheck,
    HealthCheckInterval = TimeSpan.FromSeconds(5),  // How often checks run (a group-level concern).
};

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

Use the MultiGroupOptions.HealthCheckInterval option to set the interval between health checks for all members of the group. Set this to TimeSpan.MaxValue if you want to disable periodic health checking.

The HealthCheck.Builder class provides the following properties to configure each probe:

Probe types

StackExchange.Redis provides the following built-in probe types:

Probe policies

When ProbeCount is greater than 1, the probe policy determines how the individual probe results are combined to give an overall verdict. The available policies are:

Custom health check probes

You can supply your own health check probe by deriving a new class from HealthCheckProbe, or from HealthCheckProbePolicy for a custom evaluation policy. For example, you might use this to integrate with external monitoring tools or to implement checks that are specific to your application. See the StackExchange.Redis failover documentation for details and examples.

Managing members at runtime

Although you will typically configure all members during the initial connection, you can also modify the group at runtime by casting the connection to IConnectionGroup. For example, you can add a new datacenter before decommissioning an old one for a zero-downtime migration:

var group = (IConnectionGroup)conn;

// Add a new member at runtime.
var newMember = new ConnectionGroupMember("new-dc.example.com:6379", name: "New Datacenter") { Weight = 5 };
await group.AddAsync(newMember);

// Remove a member.
group.Remove(newMember);

Each ConnectionGroupMember also exposes its current status, which you can inspect at any time using the GetMembers() method of the connection:

foreach (ConnectionGroupMember member in conn.GetMembers())
{
    Console.WriteLine($"{member.Name}: Connected={member.IsConnected}, " +
                      $"Unhealthy={member.IsUnhealthy}, Weight={member.Weight}, Latency={member.Latency}");
}

Manual failover

By default, the group selects the active member automatically based on weight and latency. However, you can also use the TryFailoverTo() method to select which member to use manually, for example to route traffic away from a region during maintenance:

// Get the members and find the one you want to fail over to.
ConnectionGroupMember target = conn.GetMembers().First(m => m.Name == "US West");

// Attempt to fail over to the specified member.
if (conn.TryFailoverTo(target))
{
    Console.WriteLine($"Successfully failed over to {target.Name}");
}

// Later, pass null to remove the explicit failover and restore automatic selection.
conn.TryFailoverTo(null);

TryFailoverTo() returns false if the target member is not connected or is not part of the group. While an explicit failover is active, the chosen member is preferred for all traffic and weight and latency are ignored. However, if the chosen member becomes unavailable, the group still falls back automatically to other connected members. Explicit failovers are not persisted across application restarts.

Monitoring failover events

You may want to take some custom action when a failover occurs. For example, you could log a warning, increment a metric, or externally persist the connection state. Use the ConnectionChanged event to react when the active member changes:

conn.ConnectionChanged += (sender, args) =>
{
    if (args.Type == GroupConnectionChangedEventArgs.ChangeType.ActiveChanged)
    {
        Console.WriteLine($"Active member changed from {args.PreviousGroup?.Name ?? "none"} to {args.Group.Name}");
    }
};

Pub/Sub and re-subscription

A connection group supports Pub/Sub messaging with automatic re-subscription to channels during failover, so you don't have to detect failovers and re-subscribe manually. When you subscribe to a channel, the subscription is established against all members (for immediate pickup during a failover), and the library filters received messages so that you only observe messages from the active member. Publishing occurs only to the active member.

Create an ISubscriber from the connection in the usual way using the GetSubscriber() method, then subscribe to one or more channels:

ISubscriber subscriber = conn.GetSubscriber();

// If a failover happens, the subscription is automatically re-established
// on the new active member.
await subscriber.SubscribeAsync(RedisChannel.Literal("notifications"), (channel, message) =>
{
    Console.WriteLine($"Received: {message}");
});

await subscriber.PublishAsync(RedisChannel.Literal("notifications"), "Hello, World!");