Develop applications with Active-Active databases

Summary: Overview of how developing applications differs for Active-Active databases from standalone Redis databases.

Overview

Developing geo-distributed, multi-master applications can be difficult.
Application developers may have to understand a large number of race
conditions between updates to various sites, network, and cluster
failures that could reorder the events and change the outcome of the
updates performed across geo-distributed writes.

Active-Active databases (formerly known as CRDB) are geo-distributed databases that span multiple Redis Software clusters.
Active-Active databases depend on [multi-master replication](https://en.wikipedia.org/wiki/Multi-master_replication) and [conflict-free
replicated data types (CRDTs)](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) to power a simple development experience
for geo-distributed applications. Active-Active databases allow developers to use existing
Redis data types and commands, but automatically handle conflicting concurrent writes to the same key
across multiple geographies. For example, developers can simply use the
`INCR` or `INCRBY` method in Redis in all instances of the geo-distributed
application, and Active-Active databases handle the additive nature of `INCR` to reflect the
correct final value.

The following example displays a sequence of events
over time: t1 to t9. This Active-Active database has two member Active-Active databases: member CRDB1 and
member CRDB2. The local operations running in each member Active-Active database are
listed under the member Active-Active database name. The sync events represent the moment
when synchronization catches up to distribute all local member Active-Active database
updates to other participating clusters and other member Active-Active databases.

|  **Time** | **Member CRDB1** | **Member CRDB2** |
|  :------: | :------: | :------: |
|  t1 | INCRBY key1 7 |  |
|  t2 |  | INCRBY key1 3 |
|  t3 | GET key1
7 | GET key1
3 |
|  t4 | — Sync — | — Sync — |
|  t5 | GET key1
10 | GET key1
10 |
|  t6 | DECRBY key1 3 |  |
|  t7 |  | INCRBY key1 6 |
|  t8 | — Sync — | — Sync — |
|  t9 | GET key1
13 | GET key1
13 |

Databases provide various approaches to address some of these concerns:

- **Active-Passive geo-distributed deployments**: With active-passive
    distributions, all writes go to an active cluster. Redis Sofware
    provides a [Replica Of]() capability that provides a similar approach.
    This can be employed when the workload is heavily balanced toward
    reads and few writes. However, WAN performance and availability
    can be unreliable, and traveling large distances for writes takes away
    from application performance and availability.
- **Two-phase commit (2PC)**: This approach is designed around a protocol
    that commits a transaction across multiple transaction managers.
    Two-phase commit provides a consistent transactional write across
    regions but fails transactions unless all participating transaction
    managers are available at the time of the transaction. The number
    of messages exchanged and its cross-regional availability
    requirement make two-phase commit unsuitable for even moderate
    throughputs and cross-geo writes that go over WANs.
- **Sync update with quorum-based writes**: This approach synchronously
    coordinates a write across the majority of replicas across
    clusters spanning multiple regions. However, just like two-phase
    commit, the number of messages exchanged and its cross-regional
    availability requirement make geo-distributed quorum writes
    unsuitable for moderate throughputs and cross-geo writes that go
    over WANs.
- **Last-Writer-Wins (LWW) conflict resolution**: Some systems provide
    simplistic conflict resolution for all types of writes where
    system clocks are used to determine the winner across conflicting
    writes. LWW is lightweight and can be suitable for simpler data.
    However, LWW can be destructive to updates that are not necessarily
    conflicting. For example, adding a new element to a set across two
    geographies concurrently would result in only one of these new
    elements appearing in the final result with LWW.
- **MVCC (multi-version concurrency control)**: MVCC systems maintain
    multiple versions of data and may expose ways for applications to
    resolve conflicts. Even though an MVCC system can provide a flexible
    way to resolve conflicting writes, it comes at the cost of great
    complexity in the development of a solution.

Even though types and commands in Active-Active databases look identical to standard Redis
types and commands, the underlying types in Redis Software are enhanced to maintain
more metadata to create the conflict-free data type experience. This
section explains what you need to know about developing with Active-Active databases on
Redis Software.

Lua scripts

Active-Active databases support Lua scripts, but unlike standard Redis, Lua scripts always
execute in effects replication mode. There is currently no way to
execute them in script-replication mode.

Eviction

The default policy for Active-Active databases is _noeviction_ mode. Redis Software version 6.0.20 and later support all eviction policies for Active-Active databases, unless [Redis Flex or Auto Tiering]() (previously known as Redis on Flash) is enabled.

For details, see [eviction for Active-Active databases (Redis Software)]() or [eviction for Active-Active databases (Redis Cloud)]().

Expiration

Expiration is supported with special multi-master semantics.

If a key's expiration time is changed at the same time on different
members of the Active-Active database, the longer extended time set via TTL on a key is
preserved.

If this command was performed on key1 on cluster #1:

[code example]

If this command was performed on key1 on cluster #2:

[code example]

The `EXPIRE` command setting the key to 50 would win.

And if this command was performed on key1 on cluster #3:

[code example]

It would win out of the three clusters hosting the Active-Active database as it sets the
TTL on key1 to an infinite time.

The replica responsible for the winning expire value is also
responsible for expiring the key and propagating a DEL effect when this
happens. From this point on, a losing replica is not responsible
for expiring the key, unless another `EXPIRE` command resets the TTL.
Furthermore, a replica that is not the owner of the expired value:

- Silently ignores the key if a user attempts to access it in READ
    mode, for example, treating it as if it was expired but not propagating a
    DEL.
- Expires it (sending a DEL) before making any modifications if a user
    attempts to access it in WRITE mode.

Expiration values are in the range of [0, 2^49] for Active-Active databases and [0, 2^64] for regular databases.

Tombstones

For conflict resolution purposes, Active-Active databases cannot immediately release a deleted key. Instead, the key is logically deleted but remains in memory as a tombstone until the garbage collector can safely remove it.

When a deleted key becomes a tombstone, it frees some memory previously consumed by the key. The size of each tombstone varies depending on the data type and the key's history.

The garbage collector automatically removes a tombstone when all instances in the Active-Active database have observed the deletion operation.

To monitor tombstones, you can use shard-level metrics exposed by [`INFO crdt`](#info) and Grafana.

Out-of-Memory (OOM) {#outofmemory-oom}

If a member Active-Active database is out of memory, that member is marked as
"inconsistent", the member stops responding to user traffic, and
the syncer initiates full reconciliation with other peers in the Active-Active database.

Active-Active database key counts

Keys are counted differently for Active-Active databases:

- DBSIZE (in `shard-cli dbsize`) reports key header instances
    that represent multiple potential values of a key before a replication conflict is resolved.
- expired_keys (in `bdb-cli info`) can be more than the keys count in DBSIZE (in `shard-cli dbsize`)
    because expires are not always removed when a key becomes a tombstone.
- The Expires average TTL (in `bdb-cli info`) is computed for local expires only.

INFO

The `INFO` command has an additional CRDT section that provides advanced troubleshooting information:

|  **Section** | **Field** | **Description** |
|  ------ | ------ | ------ |
|  **CRDT Context** | crdt_config_version | Current Active-Active database configuration version. |
|   | crdt_slots | Hash slots assigned and reported by this shard. |
|   | crdt_replid | Unique replica/shard IDs. |
|   | crdt_clock | Clock value of local vector clock. |
|   | crdt_ovc | Locally observed Active-Active database vector clock. |
|  **Peers** | | A list of currently connected peer replication peers. This is similar to the replicas list reported by Redis. |
|  **Backlogs** | | A list of peer replication backlogs currently maintained. Typically in a full mesh topology, only a single backlog is used for all peers, as the requested IDs are identical. |
|  **CRDT Stats** | crdt_sync_full | Number of inbound full synchronization processes performed. |
|   | crdt_sync_partial_ok | Number of partial (backlog based) re-synchronization processes performed. |
|   | crdt_sync_partial-err | Number of partial re-synchronization processes failed due to exhausted backlog. |
|   | crdt_merge_reqs | Number of inbound merge requests processed. |
|   | crdt_effect_reqs | Number of inbound effect requests processed. |
|   | crdt_ovc_filtered_effect_reqs | Number of inbound effect requests filtered due to old vector clock. |
|   | crdt_gc_pending | Number of elements pending garbage collection. |
|   | crdt_gc_attempted | Number of attempts to garbage collect tombstones. |
|   | crdt_gc_collected | Number of tombstones garbage collected successfully. |
|   | crdt_gc_gvc_min | The minimal globally observed vector clock, as computed locally from all received observed clocks. |
|   | crdt_stale_released_with_merge | Indicates the last stale flag transition was a result of a complete full sync. |
|  **CRDT Replicas** | | A list of crdt_replica \<uid> entries, each describes the known state of a remote instance with the following fields: |
|   | config_version | Last configuration version reported. |
|   | shards | Number of shards. |
|   | slots | Total number of hash slots. |
|   | slot_coverage | A flag indicating remote shards provide full coverage (all shards are alive). |
|   | max_ops_lag | Number of local operations not yet observed by the least updated remote shard. |
|   | min_ops_lag | Number of local operations not yet observed by the most updated remote shard. |


Active-Active Redis applications

Summary: General information to keep in mind while developing applications for an Active-Active database.

Summary: Introduction to differences in data types between standalone and Active-Active Redis databases.

Data types for Active-Active databases

Summary: How to failover your application to connect to a remote replica.

Application failover with Active-Active databases

Overview

Developing globally distributed applications can be challenging, as
developers have to think about race conditions and complex combinations
of events under geo-failovers and cross-region write conflicts. In Redis Software (RS), Active-Active databases
simplify developing such applications by directly using built-in smarts
for handling conflicting writes based on the data type in use. Instead
of depending on just simplistic "last-writer-wins" type conflict
resolution, geo-distributed Active-Active databases (formerly known as CRDBs) combine techniques defined in CRDT
(conflict-free replicated data types) research with Redis types to
provide smart and automatic conflict resolution based on the data type's
intent.

An Active-Active database is a globally distributed database that spans multiple Redis
Enterprise Software clusters. Each Active-Active database can have many Active-Active database instances
that come with added smarts for handling globally distributed writes
using the proven
[CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
approach.
[CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
research describes a set of techniques for creating systems that can
handle conflicting writes. CRDBs powered by Multi-Master Replication
(MMR) provide a straightforward and effective way to replicate your
data between regions and simplify development of complex applications
that can maintain correctness under geo-failovers and concurrent
cross-region writes to the same data.

Active-Active databases replicate data between multiple Redis Software
clusters. Common uses for Active-Active databases include disaster recovery,
geographically redundant applications, and keeping data closer to your
users' locations. MMR is always multi-directional amongst the clusters
configured in the Active-Active database. For unidirectional replication, please see the
Replica Of capabilities in Redis Software.

Example of synchronization

[Learn more about
synchronization for
each supported data type]() and [how to develop applications]() with them on Redis Software.


Application-based disaster recovery

Summary: Application-based disaster recovery for Active-Active databases using custom application-level monitoring and connectivity management.

Content

For complete control over failover and failback, you can implement disaster recovery mechanisms directly in the application server.

For more information, see [Application failover with Active-Active databases]().


Client library-based disaster recovery

Summary: Client library-based disaster recovery for Active-Active databases using Redis client libraries with built-in failover logic.

Content

Some Redis client libraries support geographic failover and failback. These client libraries monitor all Active-Active database members and instantiate connections for all endpoints in advance to allow faster failover and failback.

Advantages:

- No additional hardware or software components required.

- No high availability considerations.

- No scalability concerns.

- Tighter control over connectivity, such as timeouts, connection retries, and dynamic reconfiguration.

- OSS Cluster API support (currently only available in redis-py)

- Low latency.

Considerations:

- Requires code changes for failover and failback logic.

- Concurrent access across replicas is possible, but can be mitigated using the distributed health status provided by the database availability API requests.

- When a development framework uses Redis transparently, failover and failback might not be easy to configure.

The following diagram shows a client library-based disaster recovery approach:

The following diagram shows a client-based disaster recovery approach that also uses [connection pooling]():

For additional information, see the following client library guides for failover and failback:

- [Jedis (Java)]()

- [redis-py (Python)]()


Summary: Disaster recovery strategies for Active-Active databases using network, proxy, client library, and application-based approaches.

Summary: Network-based disaster recovery for Active-Active databases using global traffic managers and load balancing solutions.

Network-based disaster recovery

Summary: Proxy-based disaster recovery for Active-Active databases.

Proxy-based disaster recovery

Client library-based disaster recovery

Application-based disaster recovery

Overview

An application deployed with an Active-Active database connects to a database member that is geographically nearby. If that database member becomes unavailable, the application can fail over to a secondary Active-Active database member, and fail back to the original database member again if it recovers.

However, because Active-Active Redis databases do not have a built-in [failover](https://en.wikipedia.org/wiki/Failover) or failback mechanism for application connections, you must implement one of the following [disaster recovery strategies](#disaster-recovery-strategies).

Disaster recovery strategies

Depending on your requirements for Recovery Point Objective, Recovery Time Objective, consistency, scalability, resources, maintainability, and other factors, choose one of the following strategies to fail over to a secondary Active-Active member or fail back to the primary member:

- [Network-based](): Global traffic managers and load balancers handle routing at the network layer, requiring no application changes.

- [Proxy-based](): Software proxies handle detection and routing logic.

- [Client library-based](): Redis client libraries with built-in failover logic.

- [Application-based](): Custom application-level monitoring and connectivity management.

Considerations for disaster recovery

When implementing a disaster recovery strategy for an Active-Active database, consider the following:

- Is the Active-Active database an on-premise, cloud, multi-cloud, or hybrid-cloud deployment?

- Number of regions and availability zones.

- Application server redundancy and deployment locations.

- Acceptable values for the maximum amount of data that can be lost during a failure (Recovery Point Objective) and the maximum acceptable time to restore service after a failure (Recovery Time Objective).

- Latency and throughput requirements.

- Number of application errors that can be tolerated during a failure.

- Tolerance for reading stale but eventually consistent data during a failover scenario.

- Is concurrent access, in which different application servers can read from or write to different Active-Active database members, acceptable?

- Are there any regulatory or policy requirements for disaster recovery?

- Does the application connect to the Active-Active database using a Redis client library or through a development framework or ecosystem?

- Does the Active-Active database use DNS, the [OSS Cluster API](), or the [discovery service]()?

- Is rate-limiting control needed?

- Can you modify the existing codebase or introduce new components, such as load balancers or proxies?

Detect failures with health checks

Depending on the failover strategy, you can use the following health checks to detect Active-Active database failures and determine whether to fail over to a secondary Active-Active member, or fail back to the primary member after the preferred endpoint is back online.

To determine which health checks to use, consider factors such as detection speed, integrability with the failover strategy, and writability or durability guarantees.

Lag-aware database availability requests are the recommended method to detect database failures in Redis Software deployments. This method guarantees that all the shards of a clustered database are connectable.

See [Lag-aware database availability requests]() for more information.

Lag-aware database availability requests are not supported for Redis Cloud databases.

Redis connection health checks

You can use an existing connection to the database to check its availability.

#### PING command

The [`PING`]() command checks the following:

- The database is connectable.

- The database is readable.

- The dataset is available.

Example response for an available database:

[code example]

If a database is connectable but not available for reads, such as when reloading from a snapshot, `PING` returns an error message:

[code example]

#### Connection timeouts or Redis errors

By capturing connection errors, you can determine when to fail over to a secondary Active-Active member or fail back to the primary member based on the [circuit breaker pattern](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern).

#### Custom health check

You can also implement custom health checks.

For example, you can check the keyspace with write operations such as using the [`SET`]() command to write arbitrary data. This check verifies that database shards are available and writable.

For example:

[code example]

Use multiple write operations with different randomized keys to access different shards and guarantee that all the shards are available.

Health check comparison

| Health check | Connectivity | Readability | Writability | Durability | Notes |
|--------------|--------------|-------------|--------------|------------|-------|
| Database availability requests |:white_check_mark: |  |  |  | No guarantees on readability. For example, the shard might be reloading from a snapshot. |
| `PING` |:white_check_mark: |:white_check_mark: |:white_check_mark: |  | No support for clustered databases. All `PING` requests will be forwarded to shard 1. |
| Keyspace sampling |:white_check_mark: |:white_check_mark: |:white_check_mark: |:white_check_mark: | Write operations are persisted, increasing disk usage for AOF and RDB. |

127.0.0.1:6379> PING
(error) LOADING Redis is loading the dataset in memory

SET <randomized_key_name> <arbitrary_value> EX 1


Network-based disaster recovery

Overview

For cross-region availability, you can use a global traffic manager or a global load balancer. Both help route traffic across regions, but they work in different ways:

- Global traffic managers are DNS-based and don't handle traffic directly. Instead, they answer DNS queries with the best IP address to connect to, based on factors such as distance, latency, or health. After DNS resolution, the client connects directly to the endpoint.

- Global load balancers can route actual traffic and make decisions in real time, using request headers or connection state.

If your deployment does not require cross-region availability, you can use a regional load balancer to grant cross-zone redundancy.

Cross-region availability

For cross-region availability, you can use a global traffic manager or a global load balancer.

Advantages:

- If DNS routing is available at the application level, no additional load balancer is required between the application and the data tier to resolve the Active-Active database member's FQDN, reducing latency.

- Protects against data center failure since failure in one region should not affect services running in another region.

Global traffic manager

A global traffic manager acts as an intelligent DNS server that directs clients to healthy endpoints based on distance, latency, or availability. You should configure the traffic manager to route to the local region first and fail over to other regions if an issue occurs.

Advantages:

- High availability.

- Latency optimization.

- Seamless disaster recovery.

Considerations:

- DNS propagation delays affect failover time.

- DNS caches can impact proper functioning.

- Limited custom health check support.

- May route traffic during CRDT synchronization, causing stale data reads.

The following diagram shows how a global traffic manager with DNS resolution routes traffic:

If the environment does not allow DNS resolution, you can use a load balancer to direct traffic to the cluster nodes:

Global load balancer

For real-time traffic control and more advanced routing logic for cross-region failover and failback, you can use a global load balancer. However, this solution can have higher latency than a global traffic manager.

The following diagram shows how a global load balancer routes traffic between regions:

Cross-zone availability

If your deployment does not require cross-region availability, you can use a regional load balancer to route requests to a healthy Active-Active database member in a different availability zone within the same region.

The following diagram shows how a regional load balancer routes traffic across availability zones:

Advantages

Using a global traffic manager or global load balancer for disaster recovery has the following advantages:

- Because the routing logic happens at the network level:

    - No code changes are required. For applications, there is less logic to manage, and fewer working parts that can break.

    - Development frameworks are agnostic and can connect to a single member Active-Active database endpoint.

- If DNS routing is available at the application level, no additional load balancer is required between the application and the data tier to resolve the member Active-Active database’s FQDN, reducing latency.

- Failure of any kind in one region should not affect services running in another. This approach is ideal for scenarios where an entire data center is compromised, and service continuity is granted at a data center level.

Considerations

If you plan to implement network-based disaster recovery, you should also consider the following challenges:

- Standard health check mechanisms are available, but custom health checks may not be supported. Proper availability mechanisms, such as the database availability request, must be configured directly where possible or provided as an additional lightweight service alongside the application.

- Global load balancers introduce latency.

- Refreshing DNS records takes some time to propagate, which challenges the ability to control the failover time. In addition, DNS caches, often used by operating systems or the JVM, can impact the proper functioning of this approach.

- When failback is required, routing traffic from a secondary Active-Active database member to the primary database member can happen during CRDT synchronization. If the failback's target has not completed synchronization, applications might read stale data.

- The Active-Active health check policy should be designed to use the lag-aware database availability request, thus granting control over the observed CRDT replication lag. This method increases consistency in a failback scenario. However, specific global traffic managers and global load balancers might require extra effort to set the Active-Active health check policy.

- Replicating the application stack for several regions impacts planning, maintenance, and costs.


Proxy-based disaster recovery

Overview

If you add a lightweight proxy software component between the clients and the Active-Active database, applications can dynamically route requests to the optimal endpoint.

Advantages:

- Proxies provide proactive and reactive health check methods, such as polling target health periodically using either a TCP connection or an HTTP request, or monitoring live operations for errors.

- Proxies can be configured to run Active-Active health checks, such as the lag-aware database availability requests.

- If an Active-Active database member fails, a proxy can automatically detect the issue and redirect traffic to a healthy Active-Active database member without requiring DNS propagation delays or client disconnections. This enables fast, controlled failover and minimizes downtime.

Considerations:

- If you do not use DNS to resolve the Active-Active database members' FQDNs:

    - The proxies must have static IPs.

    - If you add a new node to the cluster, you must configure the proxy with the new endpoint.

    - A configuration syncer component is required to discover topology changes and reconfigure the proxy.

- Proxies introduce latency.

- Proxy failures can disconnect clients and cause disruptions.

Avoid concurrent access across replicas

If concurrent access across replicas must be avoided in every scenario, you can use a centralized proxy with a standby proxy instance for high availability.

Advantages:

- Prevents concurrent access across replicas.

- Failover and failback are simultaneous regardless of the Active-Active health check policy.

Considerations:

- Although the proxy can be monitored with a watchdog and restarted in case of failure, this setup does not grant high availability for the proxy.

- Limited scalability.

The following diagram shows a centralized proxy architecture with a standby proxy instance:

Co-locate to reduce latency and improve scalability

To reduce latency and improve scalability, you can use a proxy co-located in the application server.

Advantages:

- Reduced latency.

- Better scalability.

Considerations:

- Failover and failback might not be simultaneous depending on the Active-Active health check policy.

The following diagram shows a co-located proxy architecture where each application server has its own proxy:

Pool proxies for scalability

You can use a pool of active proxies to scale the routing layer. Application servers can balance new connections to the pool of proxies using a round-robin distribution algorithm, such as DNS-based round robin.

Advantages:

- High availability without complex monitoring and failover solutions.

- Flexible scalability of the routing layer.

Considerations:

- Concurrent access across replicas is possible, but can be mitigated using database availability API requests.

The following diagram shows a pool of proxies:


Get started with Redis Software Active-Active databases

Summary: Quick start guide to create an Active-Active database for test and development.

Overview

To get started, this article will help you set up an Active-Active database, formerly known as CRDB (conflict-free replicated database), spanning across two Redis Software
clusters for test and development environments. Here are the steps:

1. Run two Redis Software Docker containers.

1. Set up each container as a cluster.

1. Create a new Redis Software Active-Active database.

1. Test connectivity to the Active-Active database.

To run an Active-Active database on installations from the [Redis Software download package](),
set up two Redis Software installations and continue from Step 2.

This getting started guide is for development or demonstration environments.
For production environments, see [Create an Active-Active geo-replicated database]() for instructions.

Run two containers

To spin up two Redis Software containers, run these commands:

[code example]

[code example]

The `-h` option sets the hostname of the container, which is important for cluster setup and identification.

The `--name` option assigns a name to the container, making it easier to manage and reference with Docker commands.

The `-p` options map the Cluster Manager UI port (8443), REST API port (9443), and database access port differently for each container to make sure that all containers can be accessed from the host OS that is running the containers.

Set up two clusters

1. For cluster 1, go to `https://localhost:8443` in a browser on the
host machine to access the Redis Software Cluster Manager UI.

Depending on your browser, you may see a certificate error. Continue to the website.

1. Click **Create new cluster**:

1. Enter an email and password for the administrator account, then click **Next** to proceed to cluster setup:

1. Enter your cluster license key if you have one. Otherwise, a trial version is installed.

1. In the **Configuration** section of the **Cluster** settings page, enter a cluster FQDN, for example `cluster1.local`:

1. On the node setup screen, keep the default settings and click **Create cluster**:

1. Click **OK** to confirm that you are aware of the replacement of the HTTPS SSL/TLS
    certificate on the node, and proceed through the browser warning.

1. Repeat the previous steps for cluster 2 with these differences:

    - In your web browser, go to `https://localhost:8445` to set up the cluster 2.

    - For the **Cluster name (FQDN)**, enter a different name, such as `cluster2.local`.

Now you have two Redis Software clusters with FQDNs
`cluster1.local` and `cluster2.local`.

Each Active-Active instance must have a unique fully-qualified domain name (FQDN).

Create an Active-Active database

1. Sign in to cluster1.local's Cluster Manager UI at `https://localhost:8443`

1. Open the **Create database** menu with one of the following methods:

    - Click the **+** button next to **Databases** in the navigation menu:

    - Go to the **Databases** screen and select **Create database**:

1. Select **Active-Active database**.

1. Enter the cluster's local admin credentials, then click **Save**:

1. Add participating clusters that will host instances of the Active-Active database:

    1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

    1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

        In the **Other participating clusters** list, add the address and admin credentials for the other cluster: `https://cluster2.local:9443`

    1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Enter `database1` for **Database name** and `12000` for **Port**:

1. Configure additional settings:

    1. In the **High availability** section, turn off **Replication** since each cluster has only one node in this setup:

    1. In the **Clustering** section, either:

        - Make sure that **Sharding** is enabled and select the number of shards you want to have in the database. When database clustering is enabled,
        databases are subject to limitations on [Multi-key commands]().
        You can increase the number of shards in the database at any time.

        - Turn off **Sharding** to use only one shard and avoid [Multi-key command]() limitations.

You cannot enable or turn off database clustering after the Active-Active database is created.

1. Click **Create**.

1. After the Active-Active database is created, sign in to the Cluster Manager UIs for cluster 1 at `https://localhost:8443` and cluster 2 at `https://localhost:8445`.

1. Make sure each cluster has an Active-Active database member database with the name `database1`.

    In a real-world deployment, cluster 1 and cluster 2 would most likely be
    in separate data centers in different regions. However, for
    local testing we created the scale-minimized deployment using two
    local clusters running on the same host.

<!-- Also in getting-started-crdbs.md -->

Test connection

With the Redis database created, you are ready to connect to your
database. See [Connect to Active-Active databases]() for tutorials and examples of multiple connection methods.

docker run -d --cap-add sys_resource -h rs1_node1 --name rs1_node1 -p 8443:8443 -p 9443:9443 -p 12000:12000 redislabs/redis

docker run -d --cap-add sys_resource -h rs2_node1 --name rs2_node1 -p 8445:8443 -p 9445:9443 -p 12002:12000 redislabs/redis


Active-Active geo-distributed Redis

Summary: Overview of the Active-Active database in Redis Software

Summary: Information about Active-Active database to take into consideration while planning a deployment, such as compatibility, limitations, and special configuration

Considerations for planning Active-Active databases

Summary: How to create an Active-Active database and things to consider when setting it up.

Create an Active-Active geo-replicated database

Summary: How to connect to an Active-Active database using redis-cli or a sample Python application.

Connect to your Active-Active databases

Summary: Manage your Active-Active database settings.

Manage Active-Active databases

Summary: Considerations while deleting Active-Active databases.

Delete Active-Active databases

Summary: Enable causal consistency in an Active-Active database.

Enable causal consistency

Summary: How to configure distributed synchronization so that any available proxy endpoint can manage synchronization traffic.

Configure distributed synchronization

Summary: Detailed information about the syncer process and its role in distributed databases.

Syncer process

Summary: General information to keep in mind while developing applications for an Active-Active database.

Active-Active Redis applications

Overview

In Redis Software, Active-Active geo-distribution is based on [CRDT technology](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type).
The Redis Software implementation of CRDT is called an Active-Active database (formerly known as CRDB).
With Active-Active databases, applications can read and write to the same data set from different geographical locations seamlessly and with latency less than one millisecond (ms),
without changing the way the application connects to the database.

Active-Active databases also provide disaster recovery and accelerated data read-access for geographically distributed users.

High availability

The [high availability]() that Active-Active replication provides is built upon a number of Redis Software features (such as [clustering](), [replication](), and [replica HA]()) as well as some features unique to Active-Active ([multi-primary replication](), [automatic conflict resolution](), and [strong eventual consistency]()).

Clustering and replication are used together in Active-Active databases to distribute multiple copies of the dataset across multiple nodes and multiple clusters. As a result, a node or cluster is less likely to become a single point of failure. If a primary node or primary shard fails, a replica is automatically promoted to primary. To avoid having one node hold all copies of certain data, the [replica HA]() feature (enabled by default) automatically migrates replica shards to available nodes.

Multi-primary replication

In Redis Software, replication copies data from primary shards to replica shards. Active-Active geo-distributed replication also copies both primary and replica shards to other clusters. Each Active-Active database needs to span at least two clusters; these are called participating clusters.

Each participating cluster hosts an instance of your database, and each instance has its own primary node. Having multiple primary nodes means you can connect to the proxy in any of your participating clusters. Connecting to the closest cluster geographically enables near-local latency. Multi-primary replication (previously referred to as multi-master replication) also means that your users still have access to the database if one of the participating clusters fails.

Active-Active databases do not replicate the entire database, only the data.
Database configurations, LUA scripts, and other support info are not replicated.

Syncer

Keeping multiple copies of the dataset consistent across multiple clusters is no small task. To achieve consistency between participating clusters, Redis Active-Active replication uses a process called the [syncer]().

The syncer keeps a [replication backlog](), which stores changes to the dataset that the syncer sends to other participating clusters. The syncer uses partial syncs to keep replicas up to date with changes, or a full sync in the event a replica or primary is lost.

Conflict resolution

Because you can connect to any participating cluster to perform a write operation, concurrent and conflicting writes are always possible. Conflict resolution is an important part of the Active-Active technology. Active-Active databases only use [conflict-free replicated data types (CRDTs)](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). These data types provide a predictable conflict resolution and don't require any additional work from the application or client side.

When developing with CRDTs for Active-Active databases, you need to consider some important differences. See [Develop applications with Active-Active databases]() for related information.

Strong eventual consistency

Maintaining strong consistency for replicated databases comes with tradeoffs in scalability and availability. Redis Active-Active databases use a strong eventual consistency model, which means that local values may differ across replicas for short periods of time, but they all eventually converge to one consistent state. Redis uses vector clocks and the CRDT conflict resolution to strengthen consistency between replicas. You can also enable the causal consistency feature to preserve the order of operations as they are synchronized among replicas.

Other Redis Software features can also be used to enhance the performance, scalability, or durability of your Active-Active database. These include [data persistence](), [multiple active proxies](), [distributed synchronization](), [OSS Cluster API](), and [rack-zone awareness]().

Next steps

- [Plan your Active-Active deployment]()
- [Get started with Active-Active]()
- [Create an Active-Active database]()
- [Develop applications with Active-Active databases]()
- Review [disaster recovery strategies for Active-Active databases]()


Manage Active-Active databases

Overview

You can configure and manage your Active-Active database from either the Cluster Manager UI or the command line.

Database settings

Most Active-Active database settings can be changed after database creation. One notable exception is database clustering, which can't be turned on or off after the database has been created.

As of Redis Software version 8.0.16, the Cluster Manager UI supports both [global](#change-global-configuration) and [local](#change-local-configuration) configuration changes for Active-Active databases. In earlier versions, configuration changes made in the Cluster Manager UI applied only to the local instance and required additional manual updates for each participating cluster.

Change global configuration

Global configuration changes are applied to all participating clusters in the Active-Active database. However, if your Active-Active mesh contains local configurations, they will continue to override global settings on the database instances where they are applied.

Applying global configuration changes will override any local settings created using the API or earlier Cluster Manager UI versions. To keep local changes, manage them in the [**Local** configuration tab](#change-local-configuration).

**Cluster Manager UI:**

To make global configuration changes in the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. While on the **Global** tab, click **Edit**.

1. Make your configuration changes.

1. Click **Save** to apply the changes globally to all participating clusters.

**crdb-cli:**

To change the global configuration from the command line, use [`crdb-cli crdb update`]():

[code example]

Replace the placeholders `<placeholder>` with your own values.

Change local configuration

Local configuration changes override global configuration settings and are applied only to the database instance on the current cluster.

To apply changes to a local Active-Active database instance, use one of the following methods:

**Cluster Manager UI:**

To change the local configuration in the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Go to the **Local** tab.

    <img src="../../../../../images/rs/screenshots/databases/active-active-databases/local-config-tab.png" alt="The local configuration tab is selected.">

1. Click **Edit**, then make your configuration changes.

1. Click **Save** to apply the changes to the current database instance only.

**rladmin:**

To change the local configuration from the command line, use [`rladmin tune db`]():

[code example]

Replace the placeholders `<placeholder>` with your own values.

View differences between global and local configuration

In the Cluster Manager UI, an Active-Active database's **Global** configuration tab indicates when a local configuration differs from the global configuration:

If a warning symbol appears next to a setting on the **Global** configuration tab, point to the warning symbol to show details about the local configuration differences:

On the **Local** configuration tab, any locally configured settings that differ from the global settings are marked with a **Local configuration** label:

Participating clusters

You can add and remove participating clusters of an Active-Active database to change the topology.
To manage the changes to Active-Active topology, use [`crdb-cli`]() or the participating clusters list in the Cluster Manager UI.

Add participating clusters

All existing participating clusters must be online and in a syncing state when you add new participating clusters.

New participating clusters create the Active-Active database instance based on the global Active-Active database configuration.
After you add new participating clusters to an existing Active-Active database,
the new database instance can accept connections and read operations.
The new instance does not accept write operations until it is in the syncing state.

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

To add a new participating cluster to an existing Active-Active configuration using the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Click **Edit**.

1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Click **Save**.

Remove participating clusters

All existing participating clusters must be online and in a syncing state when you remove an online participating cluster.
If you must remove offline participating clusters, you can forcefully remove them.
If a forcefully removed participating cluster tries to rejoin the cluster,
its Active-Active database membership will be out of date.
The joined participating clusters reject updates sent from the removed participating cluster.
To prevent rejoin attempts, purge the forcefully removed instance from the participating cluster.

To remove a participating cluster using the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Click **Edit**.

1. In the **Participating clusters** section, point to the cluster you want to delete in the **Other participating clusters** list:

1. Click  to remove the cluster.

1. Click **Save**.

Replication backlog

Redis databases that use [replication for high availability]() maintain a replication backlog (per shard) to synchronize the primary and replica shards of a database. In addition to the database replication backlog, Active-Active databases maintain a backlog (per shard) to synchronize the database instances between clusters.

By default, both the database and Active-Active replication backlogs are set to one percent (1%) of the database size divided by the number of shards. This can range between 1MB to 250MB per shard for each backlog.

Change the replication backlog size

Use the [`crdb-cli`]() utility to control the size of the replication backlogs. You can set it to `auto` or set a specific size.

Update the database replication backlog configuration with the `crdb-cli` command shown below.

[code example]

Update the Active-Active (CRDT) replication backlog with the command shown below:

[code example]

Data persistence

Active-Active supports AOF (Append-Only File) data persistence only.  Snapshot persistence is _not_ supported for Active-Active databases and should not be used.

If an Active-Active database is currently using snapshot data persistence, use `crdb-cli` to switch to AOF persistence:
[code example]

crdb-cli crdb update --crdb-guid <guid> --<setting-name> <setting-value>

rladmin tune db { db:<id> | <name> } <setting-name> <setting-value>

crdb-cli crdb update --crdb-guid <crdb_guid> --default-db-config "{\"repl_backlog_size\": <size in MB | 'auto'>}"

crdb-cli crdb update --crdb-guid <crdb_guid> --default-db-config "{\"crdt_repl_backlog_size\": <size in MB | 'auto'>}"

crdb-cli crdb update --crdb-guid <CRDB_GUID> --default-db-config '{"data_persistence": "aof", "aof_policy":"appendfsync-every-sec"}'


Overview

In Redis Software, Active-Active geo-distribution is based on [conflict-free replicated data type (CRDT) technology](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). Compared to databases without geo-distribution, Active-Active databases have more complex replication and networking, as well as a different data type.

Because of the complexities of Active-Active databases, there are special considerations to keep in mind while planning your Active-Active database.

See [Active-Active Redis]() for more information about geo-distributed replication. For more info on other high availability features, see [Durability and high availability]().

Participating clusters

You need at least [two participating clusters]() for an Active-Active database. If your database requires more than ten participating clusters, contact Redis support. You can [add or remove participating clusters]() after database creation.

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

Changes made from the Cluster Manager UI to an Active-Active database configuration only apply to the cluster you are editing. For global configuration changes across all clusters, use the `crdb-cli` command-line utility.

Memory limits

Database memory limits define the maximum size of your database across all database replicas and [shards]() on the cluster. Your memory limit also determines the number of shards.

Besides your dataset, the memory limit must also account for replication, Active-Active metadata, and module overhead. These features can increase your database size, sometimes increasing it by two times or more.

Factors to consider when sizing your database:

- **dataset size**: you want your limit to be above your dataset size to leave room for overhead.
- **database throughput**: high throughput needs more shards, leading to a higher memory limit.
- [**modules**](): using modules with your database can consume more memory.
- [**database clustering**](): enables you to spread your data into shards across multiple nodes (scale out).
- [**database replication**](): enabling replication doubles memory consumption
- [**Active-Active replication**](): enabling Active-Active replication requires double the memory of regular replication, which can be up to two times (2x) the original data size per instance.
- [**database replication backlog**]() for synchronization between shards. By default, this is set to 1% of the database size.
- [**Active-Active replication backlog**]() for synchronization between clusters. By default, this is set to 1% of the database size.

It's also important to know Active-Active databases have a lower threshold for activating the eviction policy, because it requires propagation to all participating clusters. The eviction policy starts to evict keys when one of the Active-Active instances reaches 80% of its memory limit.

For more information on memory limits, see [Memory and performance]() or [Database memory limits]().

Networking

Network requirements for Active-Active databases include:

- A VPN between each network that hosts a cluster with an instance (if your database spans WAN).
- A network connection to [several ports](#network-ports) on each cluster from all nodes in all participating clusters.
- A [network time service](#network-time-service) running on each node in all clusters.

Networking between the clusters must be configured before creating an Active-Active database. The setup will fail if there is no connectivity between the clusters.

Network ports

Every node must have access to the REST API ports of every other node as well as other ports for proxies, VPNs, and the Cluster Manager UI. See [Network port configurations]() for more details. These ports should be allowed through firewalls that may be positioned between the clusters.

Network Time Service {#network-time-service}

Active-Active databases require a time service like NTP or Chrony to make sure the clocks on all cluster nodes are synchronized.
This is critical to avoid problems with internal cluster communications that can impact your data integrity.

See [Synchronizing cluster node clocks]() for more information.

Redis modules {#redis-modules}

Several Redis modules are compatible with Active-Active databases. Find the list of [compatible Redis modules]().

Active-Active databases created with or upgraded to Redis version 8 or later automatically enable [search and query]() and [JSON](), which allows you to index, query, and perform full-text searches of nested JSON documents.

Limitations

Active-Active databases have the following limitations:

- An existing database can't be changed into an Active-Active database. To move data from an existing database to an Active-Active database, you must [create a new Active-Active database]() and [migrate the data]().
- [Discovery service]() is not supported with Active-Active databases. Active-Active databases require FQDNs or [mDNS]().
- The `FLUSH` command is not supported from the CLI. To flush your database, use the API or Cluster Manager UI.
- The `UNLINK` command is a blocking command for all types of keys.
- Cross slot multi commands (such as `MSET`) are not supported with Active-Active databases.
- The hashing policy can't be changed after database creation.
- Active-Active databases cannot be configured as Redis Flex deployments.
- Active-Active databases handle replication internally and do not support the `redis.set_repl()` function in Lua scripts.
- If you enabled the default database password during the creation of an Active-Active database, you should not turn off the default database password because it could prevent the removal of participating database instances.


Syncer process

Syncer process

Each node in a cluster containing an instance of an Active-Active database hosts a process called the syncer.
The syncer process:

1. Connects to the proxy on another participating cluster
1. Reads data from that database instance
1. Writes the data to the local cluster's primary(master) shard

Some replication capabilities are also included in [Redis Open Source]().

The primary (also known as master) shard at the top of the primary-replica tree creates a replication ID.
This replication ID is identical for all replicas in that tree.
When a new primary is appointed, the replication ID changes, but a partial sync from the previous ID is still possible.

In a partial sync, the backlog of operations since the offset are transferred as raw operations.
In a full sync, the data from the primary is transferred to the replica as an RDB file which is followed by a partial sync.

Partial synchronization requires a backlog large enough to store the data operations until connection is restored. See [replication backlog]() for more info on changing the replication backlog size.

Syncer in Active-Active replication

In the case of an Active-Active database:

- Multiple past replication IDs and offsets are stored to allow for multiple syncs
- The [Active-Active replication backlog]() is also sent to the replica during a full sync.

Full sync triggers heavy data transfers between geo-replicated instances of an Active-Active database.

An Active-Active database uses partial synchronization in the following situations:

- Failover of primary shard to replica shard
- Restart or crash of replica shard that requires sync from primary
- Migrate replica shard to another node
- Migrate primary shard to another node as a replica using failover and replica migration
- Migrate primary shard and preserve roles using failover, replica migration, and second failover to return shard to primary

Synchronization of data from the primary shard to the replica shard is always a full synchronization.

Troubleshooting syncer errors

Unrecoverable syncer errors

Some syncer errors are unrecoverable and cause the syncer to exit with exit code 4. When this occurs, the Data Management Controller (DMC) automatically sets the `crdt_sync` or `replica_sync` value to `stopped`.

#### Restart syncer for regular databases

To restart a regular database's syncer after an unrecoverable error, [update the database configuration]() with the REST API to enable `sync`:

[code example]

#### Restart syncer for Active-Active databases

To restart an Active-Active database's syncer after an unrecoverable error, use one of the following methods.

-  For each participating cluster, [update the database configuration]() with the REST API to enable `sync`:

   [code example]

- Run [`crdb-cli crdb update`]():

   [code example]

Replace `<username>`, `<password>`, `<host>`, `<port>`, `<database-id>`, and `<crdb-guid>` with your actual values.

curl -v -k -u <username>:<password> -X PUT \
  -H "Content-Type: application/json" \
  -d '{"sync": "enabled"}' \
  https://<host>:<port>/v1/bdbs/<database-id>

crdb-cli crdb update --crdb-guid <crdb-guid> --force


Configure distributed synchronization

Overview

Replicated databases, such as [Replica Of]() and [Active-Active]() databases,
use proxy endpoints to synchronize database changes with the databases on other participating clusters.

To improve the throughput and lower the latency for synchronization traffic,
you can configure a replicated database to use distributed synchronization where any available proxy endpoint can manage synchronization traffic.

Every database by default has one proxy endpoint that manages client and synchronization communication with the database shards,
and that proxy endpoint is used for database synchronization.
This is called centralized synchronization.

To prepare a database to use distributed synchronization you must first make sure that the database [proxy policy]()
is defined so that either each node has a proxy endpoint or each primary (master) shard has a proxy endpoint.
After you have multiple proxies for the database,
you can configure the database synchronization to use distributed synchronization.

Configure distributed synchronization

You may use the database name in place of `db:<ID>` in the following `rladmin` commands.

To configure distributed synchronization:

1. To check the proxy policy for the database, run: `rladmin status`

    The output of the status command shows the list of endpoints on the cluster and the proxy policy for the endpoint.

    [code example]

    If the proxy policy (also known as a _role_) is `single`, configure the policy to `all-nodes` or `all-master-shards` according to your needs with the command:

    [code example]

1. To configure the database to use distributed synchronization, run:

    [code example]

    To change back to centralized synchronization, run:

    [code example]

Verify database synchronization

Use `rladmin` to verify a database synchronization role:

[code example]

The current database role is reported as the `syncer_mode` value:

[code example]

ENDPOINTS:
    DB:ID       NAME      ID                        NODE          ROLE                                SSL
    db:1        db        endpoint:1:1              node:1        all-master-shards                   No

rladmin bind db db:<ID> endpoint <endpoint id> policy <all-master-shards|all-nodes>

rladmin tune db db:<ID> syncer_mode distributed

rladmin tune db db:<ID> syncer_mode centralized

$ rladmin info db db:<ID>
db:<ID> [<db_name>]:
  // (Other settings removed)
  syncer_mode: centralized


Configure database persistence

Summary: How to configure database persistence with either an append-only file (AOF) or snapshots.

Overview

Data is stored in RAM or a combination of RAM and flash memory ([Redis Flex and Auto Tiering]()), which risks data loss during process or server failures. Redis Software supports multiple methods to persist data to disk on a per-database basis to ensure data durability.

You can configure [persistence](https://redis.com/redis-enterprise/technology/durable-redis/) during database creation or by editing an existing database. Although the persistence model can be changed dynamically, the switch can take time depending on the database size and the models being switched.

Configure database persistence

You can configure persistence when you [create a database](), or you can edit an existing database's configuration:

1. From the **Databases** list, select the database, then select **Configuration**.

1. Select **Edit**.

1. Expand the **Durability** section.

1. For **Persistence**, select an [option](#data-persistence-options) from the list.

1. Select **Save**.

Data persistence options

There are six options for persistence in Redis Software:

|  **Options** | **Description** |
|  ------ | ------ |
|  None | Data is not persisted to disk at all. |
|  Append-only file (AOF) - fsync every write | Data is fsynced to disk with every write. |
|  Append-only file (AOF) - fsync every 1 sec | Data is fsynced to disk every second. |
|  Snapshot, every 1 hour | A snapshot of the database is created every hour. |
|  Snapshot, every 6 hours | A snapshot of the database is created every 6 hours. |
|  Snapshot, every 12 hours | A snapshot of the database is created every 12 hours. |

Select a persistence strategy

When selecting your persistence strategy, you should take into account your tolerance for data loss and performance needs. There will always be tradeoffs between the two.
The fsync() system call syncs data from file buffers to disk. You can configure how often Redis performs an fsync() to most effectively make tradeoffs between performance and durability for your use case.
Redis supports three fsync policies: every write, every second, and disabled.

Redis also allows snapshots through RDB files for persistence. Within Redis Software, you can configure both snapshots and fsync policies.

For any high availability needs, use replication to further reduce the risk of data loss.

**For use cases where data loss has a high cost:**

Append-only file (AOF) - fsync every write - Redis Software sets the Redis directive `appendfsyncalways`.  With this policy, Redis will wait for the write and the fsync to complete prior to sending an acknowledgement to the client that the data has written. This introduces the performance overhead of the fsync in addition to the execution of the command. The fsync policy always favors durability over performance and should be used when there is a high cost for data loss.

**For use cases where data loss is tolerable only limitedly:**

Append-only file (AOF) - fsync every 1 sec - Redis will fsync any newly written data every second. This policy balances performance and durability and should be used when minimal data loss is acceptable in the event of a failure. This is the default Redis policy. This policy could result in between 1 and 2 seconds worth of data loss but on average this will be closer to one second.

If you use AOF for persistence, enable replication to improve performance. When both features are enabled for a database, the replica handles persistence, which prevents any performance impact on the master.

**For use cases where data loss is tolerable or recoverable for extended periods of time:**

- Snapshot, every 1 hour - Performs a full backup every hour.
- Snapshot, every 6 hour - Performs a full backup every 6 hours.
- Snapshot, every 12 hour - Performs a full backup every 12 hours.
- None - Does not back up or persist data at all.

Append-only file (AOF) vs snapshot (RDB)

Now that you know the available options, to assist in making a decision
on which option is right for your use case, here is a table about the
two:

|  **Append-only File (AOF)** | **Snapshot (RDB)** |
|------------|-----------------|
|  More resource intensive | Less resource intensive |
|  Provides better durability (recover the latest point in time) | Less durable |
|  Slower time to recover (Larger files) | Faster recovery time |
|  More disk space required (files tend to grow large and require compaction) | Requires less resources (I/O once every several hours and no compaction required) |

Active-Active data persistence

Active-Active databases support AOF persistence only.  Snapshot persistence is not supported for Active-Active databases.

If an Active-Active database is using snapshot persistence, use `crdb-cli` to switch to AOF persistence:

[code example]

Auto Tiering data persistence

Auto Tiering flash storage is not considered persistent storage.

Flash-based databases are expected to hold larger datasets, and shard repair times can take longer after node failures. To better protect the database against node failures with longer repair times, consider enabling master and replica dual data persistence.

However, dual data persistence with replication adds some processor
and network overhead, especially for cloud configurations
with network-attached persistent storage, such as EBS-backed
volumes in AWS.

There may be times when performance is critical for your use case and
you don't want to risk data persistence adding latency.

You can enable or turn off data persistence on the master shards using the
following `rladmin` command:

[code example]

crdb-cli crdb update --crdb-guid <CRDB_GUID> --default-db-config \
   '{"data_persistence": "aof", "aof_policy":"appendfsync-every-sec"}'

rladmin tune db <database_ID_or_name> master_persistence <disabled | enabled>


Configure database defaults

Summary: Cluster-wide policies that determine default settings when creating new databases.

Overview

Database defaults are cluster-wide policies that determine default settings when creating new databases.

Edit database defaults

To edit default database configuration using the Cluster Manager UI:

1. On the **Databases** screen, select  to open a list of additional actions.

1. Select **Database defaults**.

1. Configure [database defaults](#db-defaults).

1. Select **Save**.

Database defaults {#db-defaults}

Endpoint configuration

You can choose a predefined endpoint configuration to use the recommended database proxy and shards placement policies for your use case. If you want to set these policies manually instead, select **Custom** endpoint configuration.

| Endpoint configuration | Database proxy | Shards placement | Description |
|-----------|------------|----------------|------------------|------------|
| Enterprise clustering | Single | Dense | Sets up a single endpoint that uses DNS to automatically reflect IP address updates after failover or topology changes. |
| Using a load balancer | All nodes | Sparse | Configure Redis with a load balancer like HAProxy or Nginx for environments without DNS. |
| Multiple endpoints | All primary shards | Sparse | To set up multiple endpoints, enable **OSS Cluster API** in the database settings and ensure client support. Clients initially connect to the primary node to retrieve the cluster topology, which allows direct connections to individual Redis proxies on each node. |
| Custom | Single, all primary shards, or all nodes | Dense or sparse | Manually choose default database proxy and shards placement policies. |

Database proxy

Redis Software uses [proxies]() to manage and optimize access to database shards. Each node in the cluster runs a single proxy process, which can be active (receives incoming traffic) or passive (waits for failovers).

You can configure default [proxy policies]() to determine which nodes' proxies are active and bound to new databases by default.

To configure the default database proxy policy using the Cluster Manager UI:

1. [**Edit database defaults**](#edit-database-defaults).

1. Select a predefined [**Endpoint Configuration**](#endpoint-configuration) to use a recommended database proxy policy, or choose **Custom** to set the policy manually. Changing the database proxy default in the Cluster Manager UI affects both sharded and non-sharded proxy policies.

#### Non-sharded proxy policy

To configure the default proxy policy for non-sharded databases, use one of the following methods:

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

#### Sharded proxy policy

To configure the default proxy policy for sharded databases, use one of the following methods:

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

Shards placement

The default [shard placement policy]() determines the distribution of database shards across nodes in the cluster.

Shard placement policies include:

- `dense`: places shards on the smallest number of nodes.

- `sparse`: spreads shards across many nodes.

To configure default shard placement, use one of the following methods:

- Cluster Manager UI:

    1. [**Edit database defaults**](#edit-database-defaults).

    1. Select a predefined [**Endpoint Configuration**](#endpoint-configuration) to use a recommended shards placement policy, or choose **Custom** to set the policy manually.

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

Database version

New databases use the default Redis database version unless you select a different **Database version** when you [create a database]() in the Cluster Manager UI or specify the `redis_version` in a [create database REST API request]().

To configure the Redis database version, use one of the following methods:

- Cluster Manager UI: Edit **Database version** in [**Database defaults**](#edit-database-defaults)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

Internode encryption

Enable [internode encryption]() to encrypt data in transit between nodes for new databases by default.

To enable or turn off internode encryption by default, use one of the following methods:

- Cluster Manager UI: Edit **Internode Encryption** in [**Database defaults**](#edit-database-defaults)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

rladmin tune cluster default_non_sharded_proxy_policy { single | all-master-shards | all-nodes }

PUT /v1/cluster/policy
    { "default_non_sharded_proxy_policy": "single | all-master-shards | all-nodes" }

rladmin tune cluster default_sharded_proxy_policy { single | all-master-shards | all-nodes }

PUT /v1/cluster/policy
    { "default_sharded_proxy_policy": "single | all-master-shards | all-nodes" }

rladmin tune cluster default_shards_placement { dense | sparse }

PUT /v1/cluster/policy
    { "default_shards_placement": "dense | sparse" }

rladmin tune cluster default_redis_version <x.y>

PUT /v1/cluster/policy
    { "default_provisioned_redis_version": "x.y" }

rladmin tune cluster data_internode_encryption { enabled | disabled }

PUT /v1/cluster/policy
    { "data_internode_encryption": <boolean> }


Manage database tags

Summary: Manage tags for databases in a Redis Software cluster.

Overview

You can create custom tags to categorize databases in a Redis Software cluster.

The **Databases** screen shows tags for each database in the list.

Add database tags

You can add tags when you [create a database]() or [edit an existing database's configuration]().

To add tags to a database using the Cluster Manager UI:

1. While in edit mode on the database's configuration screen, click **Add tags**.

1. Enter a key and value for the tag. Keys and values previously used by existing tags will appear as suggestions.

1. To add additional tags, click **Add tag**.

1. After you finish adding tags, click **Done** to close the tag manager.

1. Click **Create** or **Save**.

Edit database tags

To edit a database's existing tags using the Cluster Manager UI:

1. Go to the database's **Configuration** screen, then click **Edit**.

1. Next to the existing **Tags**, click .

1. Edit or delete existing tags, or click **Add tag** to add new tags.

1. After you finish editing tags, click **Done** to close the tag manager.

1. Click **Save**.


Change database upgrade configuration

Summary: Configure cluster-wide policies that affect default database upgrades.

Overview

Database upgrade configuration includes cluster-wide policies that affect default database upgrades.

Edit upgrade configuration

To edit database upgrade configuration using the Cluster Manager UI:

1. On the **Databases** screen, select  to open a list of additional actions.

1. Select **Upgrade configuration**.

1. Change database [upgrade configuration settings](#upgrade-config-settings).

1. Select **Save**.

Upgrade configuration settings {#upgrade-config-settings}

Database shard parallel upgrade

To change the number of shards upgraded in parallel during database upgrades, use one of the following methods:

- Cluster Manager UI – Edit **Database shard parallel upgrade** in [**Upgrade configuration**](#edit-upgrade-configuration)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

RESP3 support

The cluster-wide option `resp3_default` determines the default value of the `resp3` option, which enables or deactivates RESP3 for a database, upon upgrading a database to version 7.2 or later. `resp3_default` is set to `enabled` by default.

To change `resp3_default` to `disabled`, use one of the following methods:

- Cluster Manager UI – Edit **RESP3 support** in [**Upgrade configuration**](#edit-upgrade-configuration)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

rladmin tune cluster parallel_shards_upgrade { all | <integer> }

PUT /v1/cluster/policy
    { "parallel_shards_upgrade": <integer> }

rladmin tune cluster resp3_default { enabled | disabled }

PUT /v1/cluster/policy
    { "resp3_default": <boolean> }


Configure database settings

Summary: Configure settings specific to each database.

Configure database defaults

Change database upgrade configuration

Manage database tags

Enable OSS Cluster API

Configure database persistence

Configure proxy policy

Summary: Configure high availability for replica shards so that the cluster automatically migrates the replica shards to an available node.

Summary: Configure shard placement to improve performance.

Configure shard placement

Overview

You can manage your Redis Software databases with several tools:

- [Cluster Manager UI](#edit-database-settings) (the web-based user interface)

- Command-line tools:

    - [`rladmin`]() for standalone database configuration

    - [`crdb-cli`]() for Active-Active database configuration

    - [`redis-cli`]() for Redis Open Source configuration

- [REST API]()

Edit database settings

You can change the configuration of a Redis Software database at any time.<!--more-->

To edit the configuration of a database using the Cluster Manager UI:

1. On the **Databases** screen, select the database you want to edit.

1. From the **Configuration** tab, select **Edit**.

1. Change any [configurable database settings](#config-settings).

For [Active-Active database instances](), most database settings only apply to the instance that you are editing.

1. Select **Save**.

Configuration settings {#config-settings}

General

- [**Tags**]() - Add custom tags to categorize the database.

- **Database version** - Select the Redis version when you create a database.

- **Database name** - The database name requirements are:

    - Maximum of 63 characters

    - Only letters, numbers, or hyphens (-) are valid characters

    - Must start and end with a letter or digit

    - Case-sensitive

- **Endpoint port number** - You can define the port number that clients use to connect to the database. Otherwise, a port is randomly selected.

You cannot change the [port number]()
after the database is created.

Capacity

- **Memory limit** - [Database memory limits]() include all database replicas and shards, including replica shards in database replication and database shards in database clustering.

    If the total size of the database in the cluster reaches the memory limit, the memory eviction policy for the database is enforced.

- **RAM limit** - If you create a database with Auto Tiering enabled, you also need to set the RAM-to-Flash ratio. Minimum RAM is 10%. Maximum RAM is 50%.

- [**Memory eviction**]() - By default, when the total size of the database reaches its memory limit, the database evicts keys according to the least recently used keys out of all keys with an "expire" field set to make room for new keys. You can select a different eviction policy.

Capabilities

When you create a new in-memory database, you can enable multiple Redis Stack [**Capabilities**]().

Databases created with or upgraded to Redis version 8 or later automatically enable the capabilities (modules) bundled with Redis Software as follows:

For Auto Tiering databases, you can enable capabilities that support Auto Tiering. See [Redis Software and Redis Stack feature compatibility
]() for compatibility details.

To use Redis Stack capabilities, enable them when you create a new database.
You cannot enable them after database creation.

To add capabilities to the database:

1. In the **Capabilities** section, select one or more capabilities.

1. To customize capabilities, click **Parameters** and enter the optional custom configuration.

1. Click **Done**.

To change capabilities' parameters for an existing database using the Cluster Manager UI:

  1. In the **Capabilities** section, click **Edit Parameters**.

  1. After you finish editing the module's configuration parameters, click **Done** to close the parameter editor.

High Availability

- [**Replication**]() - We recommend you use intra-cluster replication to create replica shards for each database for high availability.

    If the cluster is configured to support [rack-zone awareness](), you can also enable rack-zone awareness for the database.

- [**Replica high availability**]() - Automatically migrates replica shards to an available node if a replica node fails or is promoted to primary.

Clustering

- **Enable sharding** - You can either:

    - Turn on sharding to enable [database clustering]() and select the number of database shards.

        When database clustering is enabled, databases are subject to limitations on [Multi-key commands]().

        You can increase the number of shards in the database at any time.

    - Turn off sharding to use only one shard so that you can use [Multi-key commands]() without the limitations.

- [**Shards placement**]() - Determines how to distribute database shards across nodes in the cluster.

    - _Dense_ places shards on the smallest number of nodes.

    - _Sparse_ spreads shards across many nodes.

- [**OSS Cluster API**]() - The OSS Cluster API configuration allows access to multiple endpoints for increased throughput.

    This configuration requires clients to connect to the primary node to retrieve the cluster topology before they can connect directly to proxies on each node.

    When you enable the OSS Cluster API, shard placement changes to _Sparse_, and the database proxy policy changes to _All primary shards_ automatically.

You must use a client that supports the cluster API to connect to a database that has the cluster API enabled.

- **Hashing policy** - You can accept the [standard hashing policy](), which is compatible with Redis Open Source, or define a [custom hashing policy]() to define where keys are located in the clustered database.

- [**Database proxy**]() - Determines the number and location of active proxies, which manage incoming database operation requests.

Durability

- [**Persistence**]() - To protect against loss of data stored in RAM, you can enable data persistence and store a copy of the data on disk with snapshots or an append-only file.

- **Scheduled backup** - You can configure [periodic backups]() of the database, including the interval and backup location parameters.

TLS

You can require [**TLS**]() encryption and authentication for all communications, TLS encryption and authentication for Replica Of communication only, and TLS authentication for clients.

Access control

- **Unauthenticated access** - You can access the database as the default user without providing credentials.

- **Password-only authentication** - When you configure a password for your database's default user, all connections to the database must authenticate with the [AUTH command]().

    If you also configure an access control list, connections can specify other users for authentication, and requests are allowed according to the Redis ACLs specified for that user.

    Creating a database without ACLs enables a *default* user with full access to the database. You can secure default user access by requiring a password.

- **Access Control List** - You can specify the [user roles]() that have access to the database and the [Redis ACLs]() that apply to those connections.

    To define an access control list for a database:

    1. In **Security > Access Control > Access Control List**, select **+ Add ACL**.

    1. Select a [role]() to grant database access.

    1. Associate a [Redis ACL]() with the role and database.

    1. Select the check mark to add the ACL.

Alerts

Select [alerts]() to show in the database status and configure their thresholds.

You can also choose to [send alerts by email]() to relevant users.

Replica Of

With [**Replica Of**](), you can make the database a repository for keys from other databases.

RESP3 support

[RESP]()  (Redis Serialization Protocol) is the protocol clients use to communicate with Redis databases. If you enable RESP3 support, the database will support the RESP3 protocol in addition to RESP2.

For more information about Redis Software's compatibility with RESP3, see [RESP compatibility with Redis Software]().

Internode encryption

Enable **Internode encryption** to encrypt data in transit between nodes for this database. See [Internode encryption]() for more information.


Enable OSS Cluster API

Overview

Review [OSS Cluster API]() to determine if you should enable this feature for your database.

Prerequisites

The OSS Cluster API is supported only when a database meets specific criteria.

The database must:

- Use the standard [hashing policy]().
- Have the [proxy policy]() set to either _All primary shards_ or _All nodes_.

In addition, the database must _not_:

- Use node `include` or `exclude` in the proxy policy.
- Use [RedisTimeSeries]() or [RedisGears]() modules.

The OSS Cluster API setting applies to individual databases instead of the entire cluster.

For Kubernetes deployments, see [Enable cluster-aware clients (OSS Cluster API)]().

Enable OSS Cluster API support

You can use the Cluster Manager UI, the `rladmin` utility, or the REST API to enable OSS Cluster API support for a database.

When you enable OSS Cluster API support for an existing database, the change applies to new connections but does not affect existing connections. Clients must close existing connections and reconnect to apply the change.

**Cluster Manager UI:**

When you use the Cluster Manager UI to enable the OSS Cluster API, it automatically configures the [prerequisites]().

To enable the OSS Cluster API for an existing database in the Cluster Manager UI:

1. From the database's **Configuration** tab, click **Edit**.

1. Expand the **Clustering** section.

1. Enable **Sharding**.

1. Select **OSS Cluster API**.

    <img src="../../../../../images/rs/screenshots/databases/config-clustering-oss-cluster-api-7-22.png" alt="Use the *OSS Cluster API* setting to enable the API for the selected database.">

1. Click **Save**.

You can also use the Cluster Manager UI to enable the setting when creating a new database.

**rladmin:**

You can use the [`rladmin` utility]() to enable the OSS Cluster API for Redis Software databases, including Replica Of databases.

For Active-Active (CRDB) databases, [use the crdb-cli utility](#active-active-databases).

Ensure the [prerequisites]() have been configured.  Then, enable the OSS Cluster API for a Redis database from the command line:

[code example]

To determine the current setting for a database from the command line, use `rladmin info db` to return the value of the `oss_cluster` setting.

[code example]

The OSS Cluster API setting applies to the specified database only; it does not apply to the cluster.

**REST API:**

You can enable the OSS Cluster API when you [create a database]() using the REST API:

[code example]

To enable the OSS Cluster API for an existing database, you can use an [update database configuration]() REST API request:

[code example]

Active-Active databases

The OSS Cluster API setting applies to all instances of the Active-Active database across participating clusters. To enable the OSS Cluster API for Active-Active databases, use the [Cluster Manager UI](#cluster-manager-ui) or the [`crdb-cli`]() utility.

**Cluster Manager UI:**

When you use the Cluster Manager UI to enable the OSS Cluster API, it automatically configures the [prerequisites]().

To enable the OSS Cluster API for an existing Active-Active database in the Cluster Manager UI:

1. From the database's **Configuration** tab, click **Edit**.

1. Expand the **Clustering** section.

1. Sharding must be enabled during Active-Active database creation.

1. Select **OSS Cluster API**.

    <img src="../../../../../images/rs/screenshots/databases/config-clustering-oss-cluster-api-active-active-7-22.png" alt="Use the *OSS Cluster API* setting to enable the API for the selected database.">

1. Click **Save**.

You can also use the Cluster Manager UI to enable the setting when creating a new database.

**crdb-cli:**

To create an Active-Active database and enable the OSS Cluster API with `crdb-cli`:

[code example]

See the [`crdb-cli crdb create`]() reference for more options.

To enable the OSS Cluster API for an existing Active-Active database with `crdb-cli`:

1. Obtain the `CRDB-GUID` for the new database:

    [code example]

1. Use the `CRDB-GUID` to enable the OSS Cluster API:

    [code example]

Change preferred IP type

By default, using [`CLUSTER SLOTS`]() and [`CLUSTER SHARDS`]() in a Redis Software cluster exposes the internal IP addresses for databases with the OSS Cluster API enabled.

To use external IP addresses instead of internal IP addresses, run the following [`rladmin tune db`]() command for each affected database:

[code example]

Turn off OSS Cluster API support

To deactivate OSS Cluster API support for a database, use one of the following methods:

**Cluster Manager UI:**

To turn off the OSS Cluster API for standard databases or Active-Active databases in the Cluster Manager UI:

1. From the database's **Configuration** tab, click **Edit**.

1. Expand the **Clustering** section.

1. Clear the **OSS Cluster API** checkbox.

1. Click **Save**.

**rladmin:**

For standard databases, including Replica Of, you can use `rladmin` to turn off the OSS Cluster API:

[code example]

**crdb-cli:**

For Active-Active databases, you can use `crdb-cli` to turn off the OSS Cluster API:

[code example]

When you turn off OSS Cluster API support for an existing database, the change applies to new connections but does not affect existing connections. Clients must close existing connections and reconnect to apply the change.

Multi-key command support

When you enable the OSS Cluster API for a database,
[multi-key commands]() are only allowed when all keys are mapped to the same slot.

To verify that your database meets this requirement, make sure that the `CLUSTER KEYSLOT` reply is the same for all keys affected by the multi-key command.  To learn more, see [multi-key operations]().

$ rladmin tune db <database name or ID> oss_cluster enabled

$ rladmin info db test | grep oss_cluster:
  oss_cluster: enabled

POST /v1/bdbs
{
  "oss_cluster": true,
  // Other database configuration parameters
}

PUT /v1/bdbs/<database-id>
{ "oss_cluster": true }

$ crdb-cli crdb create --name <name> \
    --memory-size 10g --port <port> \
    --sharding true --shards-count 2  \
    --replication true --oss-cluster true --proxy-policy all-master-shards \
    --instance fqdn=<fqdn>,username=<user>,password=<pass> \
    --instance fqdn=<fqdn>,username=<user>,password=<pass> \
    --instance fqdn=<fqdn>,username=<user>,password=<pass>

$ crdb-cli crdb list
    CRDB-GUID    NAME   REPL-ID  CLUSTER-FQDN
    <CRDB-GUID>  Test   4        cluster1.local

$ crdb-cli crdb update --crdb-guid <CRDB-GUID> \
        --oss-cluster true

$ rladmin tune db db:<database-id> oss_cluster_api_preferred_ip_type external

$ rladmin tune db <name or ID> oss_cluster disabled

$ crdb-cli crdb update --crdb-guid <CRDB-GUID> \
        --oss-cluster false


Configure proxy policy

Overview

Redis Software provides high-performance data access
through a proxy process that manages and optimizes access to shards
within the Redis Software cluster. Each node contains a single proxy process.
Each proxy can be active and take incoming traffic or it can be passive
and wait for failovers.

Proxy policies

A database can have one of the following proxy policies:

| Proxy policy | Description | Recommended use cases | Advantages | Disadvantages |
|--------------|-------------|-----------------------|-----------|-----------------|
| Single | Only a single proxy is bound to the database. This is the default database configuration. | Most use cases without high traffic or load | Lower resource usage, fewer application-to-cluster connections | Higher latency, more network hops |
| All primary shards | Multiple proxies are bound to the database, one on each node that hosts a database primary shard. | Most use cases that require multiple endpoints, such as when using the [OSS Cluster API]() | Lower latency, fewer network hops, higher throughput | Higher resource usage, more application-to-proxy connections |
| All nodes | Multiple proxies are bound to the database, one on each node in the cluster, regardless of whether or not there is a shard from this database on the node. | When using [load balancers]() for environments without DNS | Higher throughput | Highest resource usage |

View proxy policy

You can use the Cluster Manager UI, [`rladmin`](), or the [REST API]() to view proxy configuration settings.

The [`rladmin info cluster`]() command returns the current proxy policy for sharded and non-sharded (single shard) databases.

[code example]

Configure database proxy policy

You can use the [Cluster Manager UI](#cluster-manager-ui-method), the [REST API](#rest-api-method), or [`rladmin`](#command-line-method) to configure a database's proxy policy.

Any configuration update that unbinds existing proxies can disconnect existing client connections.

Cluster Manager UI method

You can change a database's proxy policy when you [create]() or [edit]() a database using the Cluster Manager UI:

1. While in edit mode on the database's configuration screen, expand the **Clustering** section.

1. Select a policy from the **Database proxy** list.

1. Click **Create** or **Save**.

REST API method

You can specify a proxy policy when you [create a database]() using the REST API:

[code example]

To change the proxy policy of an existing database and endpoint, you can use an [update database configuration]() REST API request:

[code example]

Command-line method

You can configure a database's proxy policy using [`rladmin bind`]().

The following example changes the bind policy for a database named "db1" with an endpoint ID "1:1" to "All primary shards" proxy policy:

[code example]

The next command performs the same task using the database ID instead of the name. The ID of this database is "1".

[code example]

You can find the endpoint ID for the endpoint argument by running `rladmin status`. Look for the endpoint ID information under
the `ENDPOINT` section of the output.

Reapply policies after topology changes

If you want to reapply the policy after topology changes, such as node restarts,
failovers and migrations, run this command to reset the policy:

[code example]

This is not required with single policies.

#### Other implications

During the regular operation of the cluster different actions might take
place, such as automatic migration or automatic failover, which change
what proxy needs to be bound to what database. When such actions take
place the cluster attempts, as much as possible, to automatically change
proxy bindings to adhere to the defined policies. That said, the cluster
attempts to prevent any existing client connections from being
disconnected, and hence might not entirely enforce the policies. In such
cases, you can enforce the policy using the appropriate rladmin
commands.

Multiple active proxies

Each database you create in a Redis Software cluster has an endpoint, which consists of a unique URL and port on the FQDN. This endpoint receives all the traffic for all operations for that database. By default, Redis Software binds this database endpoint to one of the proxies on a single node in the cluster. This proxy becomes an active proxy and receives all the operations for the given database. If the node with the active proxy fails, a new proxy on another node takes over as part of the failover process automatically.

In most cases, a single proxy can handle a large number of operations
without consuming additional resources. However, under high load,
network bandwidth, or a high rate of packets per second (PPS) on the
single active proxy can become a bottleneck to how fast database
operations can be performed. In such cases, having multiple active proxies across multiple nodes, mapped to the same external database
endpoint, can significantly improve throughput.

You can configure a database to have multiple internal proxies, which can improve performance in some cases.
Even though multiple active proxies can help improve the throughput of database
operations, configuring multiple active proxies may cause additional
latency in operations as the shards and proxies are spread across
multiple nodes in the cluster.

When the network on a single active proxy becomes the bottleneck, consider enabling multiple NIC support in Redis Software. With nodes that have multiple physical NICs (Network Interface Cards), you can configure Redis Software to separate internal and external traffic onto independent physical NICs. For more details, refer to [Multi-IP & IPv6]().

Having multiple proxies for a database can improve Redis Software's ability for fast failover in case of proxy or node failure. With multiple proxies for a database, a client doesn't need to wait for the cluster to spin up another proxy and a DNS change in most cases. Instead, the client uses the next IP address in the list to connect to another proxy.

$ rladmin info cluster
cluster configuration:
   ...
   default_non_sharded_proxy_policy: single
   default_sharded_proxy_policy: single
   ...

POST /v1/bdbs
{
  "proxy_policy": "single | all-master-shards | all-nodes",
  // Other database configuration parameters
}

PUT /v1/bdbs/<database-id>
{
  "proxy_policy": "single | all-master-shards | all-nodes",
  "endpoint": <endpoint_uid>
}

rladmin bind db db1 endpoint 1:1 policy all-master-shards

rladmin bind db db:1 endpoint 1:1 policy all-master-shards

rladmin bind db db:<ID> endpoint <endpoint id> policy <all-master-shards|all-nodes>