Stay up when a region goes down: Highly available Redis for Python apps

Active-Active Redis & client-side geographic failover

Active-Active Redis distributes data across multiple regions, allowing each regional database instance to serve both reads and writes. What kind of magic allows that? Redis uses conflict-free replicated data types (CRDTs) to resolve concurrent updates and ensure that the instances eventually converge to a consistent state.

A standard redis-py client connects to a single configured endpoint. In order to be able to quickly fail over between instances, in case of a failure, an application could create separate clients for multiple regional database instances, but it must then implement (and maintain!) health monitoring, endpoint selection, failover, and failback. Having countless applications around the world implementing the same logic, some with more success than others, doesn’t really make a lot of engineering sense, so we decided to come up with a single “canonical” implementation which provides a client API that manages these responsibilities, thereby enabling client-side geographic failover.

This API is exposed on MultiDBClient - a wrapper over the regular single and cluster client instances. The MultiDBClient routes traffic to one selected endpoint (active database) - while monitoring the health of all configured endpoints. If the active database is considered unhealthy, the client selects another healthy endpoint according to the configured weights and redirects traffic to it. When automatic failback is enabled, the client periodically evaluates the unavailable endpoints and can return to the highest-weighted healthy endpoint. MultiDBClient does not broadcast commands or replicate data across the endpoints. Data replication remains the responsibility of the Active-Active database layer letting the CRDTs shine.

Inside MultiDBClient

Each DatabaseConfig defines one endpoint and its weight. Create one for every regional database instance. MultiDbConfig collects these endpoint configurations and defines behavior that applies to the overall MultiDBClient setup, including health checks, retries, failover, and failback. As we’ll see later, there are plenty of knobs to allow for different scenarios and setups.

MultiDBClient delegates endpoint communication and connection management to an underlying Redis or RedisCluster client. You can select tone or the other globally with MultiDbConfig.client_class: use Redis for standard endpoints and RedisCluster for endpoints exposing the OSS Cluster API. Pass endpoint-specific client options through DatabaseConfig.client_kwargs.

From failure detection to recovery

The MultiDBClient uses a circuit-breaker pattern to control traffic to each endpoint. When the active endpoint is considered unhealthy, its circuit opens and the client fails over to the highest-weighted healthy endpoint. The concept comes from electric circles, where circuit breaker elements protect an electric circle from the damage of the excess of what the equipment can actually carry.

The circuit breaker is only one of two complementary mechanisms that detect failures. The proactive background health check periodically evaluates every endpoint using the configured health-check policy. The reactive FailureDetector we described above, observes command successes and failures within a sliding window and opens the circuit when the configured failure thresholds are reached. Together, they allow the client to detect and respond to failures more quickly.

When the active endpoint’s circuit opens, the failover strategy selects the highest-weighted healthy endpoint and routes subsequent commands to it. It’s worth noting that commands already in flight against the previous endpoint cannot be redirected. If an in-flight write succeeds there, its result might not be immediately visible through the new endpoint until Active-Active replication catches up.

Eligible command failures are handled by the global retry policy. Before each retry, MultiDBClient checks the currently active endpoint, allowing the command to be retried against the newly selected database.

Automatic failback periodically checks whether a higher-weighted endpoint has recovered. Once the endpoint is healthy, MultiDBClient can switch traffic back to it. Weights can be configured to prioritize the endpoint closest to the application.

This system would allow you to always prefer endpoints closest to the application, while being prepared for outages.

For more control, you can disable automatic failback by setting auto_fallback_interval to -1 and dynamically selecting a healthy endpoint explicitly with set_active_database().

Configuring highly-available Python client

Health-check policies

A health check can run several probes before deciding whether an endpoint is healthy. The policy determines how their results are combined, balancing fast failover against tolerance for transient failures:

• HEALTHY_ALL - strict: The endpoint is healthy only when every probe succeeds. Use it when continuing to send traffic to an unstable endpoint is riskier than an occasional unnecessary failover. Its downside is low tolerance for transient network errors and a longer evaluation time.
• HEALTHY_ANY - permissive: The endpoint remains healthy when at least one probe succeeds, and probing stops after the first success. Use it when brief connection failures are expected or failover is expensive. It minimizes false-positive failovers but may keep traffic on a degraded endpoint and takes all configured probes to confirm a complete outage.
• HEALTHY_MAJORITY - balanced: More than half of the probes must succeed. It tolerates occasional failures while still responding to persistent problems. This is the best starting point for most production deployments.

In short: choose HEALTHY_ALL to prioritize endpoint quality, HEALTHY_ANY to prioritize stability and avoid unnecessary switching, or HEALTHY_MAJORITY when you need a balance between the two.

For example, the following configuration runs five probes and considers an endpoint healthy when at least three succeed:

How the settings affect performance

Health-check interval: Frequent checks improve failure detection when application traffic is absent, but every application instance runs them, which sometimes can be too wasteful.

For high-traffic systems, let the reactive detector lead and use health checks as a slower safety net. For low-traffic systems, use more frequent health checks because the reactive detector may not receive enough commands.

Failure-detection window: A short window reacts quickly and forgets old failures sooner, which suits high traffic. Low-traffic applications need a longer window; otherwise they may never collect enough samples to reach min_num_failures.

Failover attempts: failover_attempts × failover_delay defines the approximate recovery window when every endpoint is temporarily unavailable. A latency-sensitive API should keep this window short and return control to the application. A background worker can wait longer for an endpoint to recover.

High-throughput scenario

This preset favors throughput and fast failure detection:

• Short timeouts prevent requests from occupying connections for too long.
• One retry limits traffic amplification during an outage.
• Jitter prevents all application instances from retrying simultaneously.
• The short failure-detection window reacts quickly to concentrated failures.
• A longer health-check interval limits background traffic.

Observability and application integration

MultiDBClient provides runtime notifications whenever failover or failback changes the active database. Applications can use custom event listeners to record the transition, update metrics, trigger alerts, or synchronize external state. These listeners are registered through an EventDispatcher supplied to MultiDbConfig.

When redis-py observability is enabled, MultiDBClient records the redis.client.geofailover.failovers counter. It includes the attributes:

• db.client.geofailover.fail_from
• db.client.geofailover.fail_to
• db.client.geofailover.reason, such as automatic or manual

Regional failover demonstration

To see how MultiDBClient handles regional failover and failback end to end, explore the redis-py geographic failover example.

Conclusion

Active-Active Redis takes care of syncing your data across regions, while MultiDBClient helps your Python application stay connected. It brings health monitoring, weighted endpoint selection, failover and failback, retries, and observability together behind a familiar Redis client API.

There’s no one-size-fits-all configuration for high availability. The defaults offer a balanced starting point, but you should still tune them for your workload, expected failover speed, and tolerance for false positives.

So, when a region goes down, your Python application has a clear path to keep running.