title: Sorted sets in Active-Active databases

title: Content

text:
[Redis Geospatial (Geo)]() is based on Sorted Sets, so the same Active-Active database development instructions apply to Geo.

Similar to Redis Sets, Redis Sorted Sets are non-repeating collections
of Strings. The difference between the two is that every member of a
Sorted Set is associated with a score used to order the Sorted Set from
lowest to highest. While members are unique, they may have the same
score.

With Sorted Sets, you can quickly add, remove or update elements as
well as get ranges by score or by rank (position). Sorted Sets in Active-Active databases
behave the same and maintain additional metadata to handle concurrent
conflicting writes. Conflict resolution is done in two
phases:

1. First, the database resolves conflict at the set level using "OR
    Set" (Observed-Remove Set). With OR-Set behavior, writes across
    multiple Active-Active database instances are typically unioned except in cases of
    conflicts. Conflicting writes can happen when an Active-Active database instance
    deletes an element while the other adds or updates the same element.
    In this case, an observed Remove rule is followed, and only
    instances it has already seen are removed. In all other cases, the
    Add / Update element wins.
1. Second, the database resolves conflict at the score level. In this
    case, the score is treated as a counter and applies the same
    conflict resolution as regular counters.

See the following examples to get familiar with Sorted Sets'
behavior in Active-Active database:

Example of Simple Sorted Set with No
Conflict:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | ZADD Z 1.2 y |
|  t4 | — Sync — | — Sync — |
|  t5 | ZRANGE Z 0 -1 => x y | ZRANGE Z 0 -1 => x y |

**Explanation**:
When adding two different elements to a Sorted Set from different
replicas (in this example, x with score 1.1 was added by Instance 1 to
Sorted Set Z, and y with score 1.2 was added by Instance 2 to Sorted Set
Z) in a non-concurrent manner (i.e. each operation happened separately
and after both instances were in sync), the end result is a Sorted
Set including both elements in each Active-Active database instance.
Example of Sorted Set and Concurrent
Add:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 |  | ZADD Z 2.1 x |
|  t3 | ZSCORE Z x => 1.1 | ZSCORE Z x => 2.1 |
|  t4 | — Sync — | — Sync — |
|  t5 | ZSCORE Z x => 2.1 | ZSCORE Z x => 2.1 |

**Explanation**:
When concurrently adding an element x to a Sorted Set Z by two different
Active-Active database instances (Instance 1 added score 1.1 and Instance 2 added score
2.1), the Active-Active database implements Last Write Win (LWW) to determine the score of
x. In this scenario, Instance 2 performed the ZADD operation at time
t2\>t1 and therefore the Active-Active database sets the score 2.1 to
x.

Example of Sorted Set with Concurrent Add Happening at the Exact Same
Time:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x | ZADD Z 2.1 x |
|  t2 | ZSCORE Z x => 1.1 | ZSCORE Z x => 2.1 |
|  t3 | — Sync — | — Sync — |
|  t4 | ZSCORE Z x => 1.1 | ZSCORE Z x => 1.1 |

**Explanation**:
The example above shows a relatively rare situation, in which two Active-Active database
instances concurrently added the same element x to a Sorted Set at the
same exact time but with a different score, i.e. Instance 1 added x with
a 1.1 score and Instance 2 added x with a 2.1 score. After syncing, the
Active-Active database realized that both operations happened at the same time and
resolved the conflict by arbitrarily (but consistently across all Active-Active database
instances) giving precedence to Instance 1.
Example of Sorted Set with Concurrent Counter
Increment:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | ZINCRBY Z 1.0 x | ZINCRBY Z 1.0 x |
|  t4 | — Sync — | — Sync — |
|  t5 | ZSCORE Z x => 3.1 | ZSCORE Z x => 3.1 |

**Explanation**:
The result is the sum of all
ZINCRBY
operations performed by all Active-Active database instances.

Example of Removing an Element from a Sorted
Set:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 4.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | ZSCORE Z x => 4.1 | ZSCORE Z x => 4.1 |
|  t4 | ZREM Z x | ZINCRBY Z 2.0 x |
|  t5 | ZSCORE Z x => nill | ZSCORE Z x => 6.1 |
|  t6 | — Sync — | — Sync — |
|  t7 | ZSCORE Z x => 2.0 | ZSCORE Z x => 2.0 |

**Explanation**:
At t4 - t5, concurrent ZREM and ZINCRBY operations ran on Instance 1
and Instance 2 respectively. Before the instances were in sync, the ZREM
operation could only delete what had been seen by Instance 1, so
Instance 2 was not affected. Therefore, the ZSCORE operation shows the
local effect on x. At t7, after both instances were in-sync, the Active-Active database
resolved the conflict by subtracting 4.1 (the value of element x in
Instance 1) from 6.1 (the value of element x in Instance 2).


title: Streams in Active-Active databases

summary: Information about using streams with an Active-Active database.

page_type: content

title: Overview

role: overview

text:
A [Redis Stream]() is a data structure that acts like an append-only log.
Each stream entry consists of:

- A unique, monotonically increasing ID
- A payload consisting of a series key-value pairs

You add entries to a stream with the XADD command. You access stream entries using the XRANGE, XREADGROUP, and XREAD commands (however, see the caveat about XREAD below).

title: Streams and Active-Active

role: content

text:
Active-Active databases allow you to write to the same logical stream from more than one region.
Streams are synchronized across the regions of an Active-Active database.

In the example below, we write to a stream concurrently from two regions. Notice that after syncing, both regions have identical streams:

Time
Region 1
Region 2

t1
XADD messages * text hello
XADD messages * text goodbye

t2
XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929246795-2]

t3
— Sync —
— Sync —

t4
XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]
XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]

Notice also that the synchronized streams contain no duplicate IDs. As long as you allow the database to generate your stream IDs, you'll never have more than one stream entry with the same ID.

Redis Open Source uses one radix tree (referred to as `rax` in the code base) to implement each stream. However, Active-Active databases implement a single logical stream using one `rax` per region.
Each region adds entries only to its associated `rax` (but can remove entries from all `rax` trees).
This means that XREAD and XREADGROUP iterate simultaneously over all `rax` trees and return the appropriate entry by comparing the entry IDs from each `rax`.

title: Conflict resolution

text:
Active-Active databases use an "observed-remove" approach to automatically resolve potential conflicts.

With this approach, a delete only affects the locally observable data.

In the example below, a stream, `x`, is created at _t1_. At _t3_, the stream exists in two regions.

Time
Region 1
Region 2

t1
XADD messages * text hello

t2
— Sync —
— Sync —

t3
XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929244828-1]

t4
DEL messages
XADD messages * text goodbye

t5
— Sync —
— Sync —

t6
XRANGE messages - +
→ [1589929246795-2]
XRANGE messages - +
→ [1589929246795-2]

At _t4_, the stream is deleted from Region 1. At the same time, an entry with ID ending in `3700` is added to the same stream at Region 2. After the sync, at _t6_, the entry with ID ending in `3700` exists in both regions. This is because that entry was not visible when the local stream was deleted at _t4_.

title: ID generation modes

text:
Usually, you should allow Redis streams generate its own stream entry IDs. You do this by specifying `*` as the ID in calls to XADD. However, you _can_ provide your own custom ID when adding entries to a stream.

Because Active-Active databases replicate asynchronously, providing your own IDs can create streams with duplicate IDs. This can occur when you write to the same stream from multiple regions.

| Time | Region 1                      | Region 2                      |
| ---- | ------------------------------- | ------------------------------- |
| _t1_   | `XADD x 100-1 f1 v1`              | `XADD x 100-1 f1 v1`              |
| _t2_   | _— Sync —_                    | _— Sync —_                    |
| _t3_   | `XRANGE x - +`
 **→ [100-1, 100-1]** | `XRANGE x - +`
 **→ [100-1, 100-1]** |

In this scenario, two entries with the ID `100-1` are added at _t1_. After syncing, the stream `x` contains two entries with the same ID.

Stream IDs in Redis Open Source consist of two integers separated by a dash ('-'). When the server generates the ID, the first integer is the current time in milliseconds, and the second integer is a sequence number. So, the format for stream IDs is MS-SEQ.

To prevent duplicate IDs and to comply with the original Redis streams design, Active-Active databases provide three ID modes for XADD:

1. **Strict**: In _strict_ mode, XADD allows server-generated IDs (using the '`*`' ID specifier) or IDs consisting only of the millisecond (MS) portion. When the millisecond portion of the ID is provided, the ID's sequence number is calculated using the database's region ID. This prevents duplicate IDs in the stream. Strict mode rejects full IDs (that is, IDs containing both milliseconds and a sequence number).
1. **Semi-strict**: _Semi-strict_ mode is just like _strict_ mode except that it allows full IDs (MS-SEQ). Because it allows full IDs, duplicate IDs are possible in this mode.
1. **Liberal**: XADD allows any monotonically ascending ID. When given the millisecond portion of the ID, the sequence number will be set to `0`. This mode may also lead to duplicate IDs.

The default and recommended mode is _strict_, which prevents duplicate IDs.

Why do you want to prevent duplicate IDs? First, XDEL, XCLAIM, and other commands can affect more than one entry when duplicate IDs are present in a stream. Second, duplicate entries may be removed if a database is exported or renamed.

To change XADD's ID generation mode, use the `rladmin` command-line utility:

Set _strict_ mode:
[code example]

Set _semi-strict_ mode:
[code example]

Set _liberal_ mode:
[code example]

title: Iterating a stream with XREAD

text:
In Redis Open Source and in non-Active-Active databases, you can use XREAD to iterate over the entries in a Redis Stream. However, with an Active-Active database, XREAD may skip entries. This can happen when multiple regions write to the same stream.

In the example below, XREAD skips entry `115-2`.

| Time | Region 1                                         | Region 2                                         |
| ---- | -------------------------------------------------- | -------------------------------------------------- |
| _t1_   | `XADD x 110 f1 v1`                                   | `XADD x 115 f1 v1`                                   |
| _t2_   | `XADD x 120 f1 v1`                                   |                                                    |
| _t3_   | `XADD x 130 f1 v1`                                   |                                                    |
| _t4_   | `XREAD COUNT 2 STREAMS x 0`
**→ [110-1, 120-1]**       |                                                    |
| _t5_   | _— Sync —_                                       | _— Sync —_                                       |
| _t6_   | `XREAD COUNT 2 STREAMS x 120-1`
**→ [130-1]**            |                                                    |
| _t7_   | `XREAD STREAMS x 0`
**→[110-1, 115-2, 120-1, 130-1]** | `XREAD STREAMS x 0`
**→[110-1, 115-2, 120-1, 130-1]** |

You can use XREAD to reliably consume a stream only if all writes to the stream originate from a single region. Otherwise, you should use XREADGROUP, which always guarantees reliable stream consumption.

title: Consumer groups

text:
Active-Active databases fully support consumer groups with Redis Streams. Here is an example of creating two consumer groups concurrently:

| Time | Region 1                  | Region 2                  |
| ---- | --------------------------- | --------------------------- |
| _t1_   | `XGROUP CREATE x group1 0`        | `XGROUP CREATE x group2 0`        |
| _t2_   | `XINFO GROUPS x`
**→ [group1]**     | `XINFO GROUPS x`
**→ [group2]**     |
| _t3_   | _— Sync —_                | — Sync —                |
| _t4_   | `XINFO GROUPS x`
**→ [group1, group2]** | `XINFO GROUPS x`
**→ [group1, group2]** |

Redis Open Source uses one radix tree (`rax`) to hold the global pending entries list and another `rax` for each consumer's PEL.
The global PEL is a unification of all consumer PELs, which are disjoint.

An Active-Active database stream maintains a global PEL and a per-consumer PEL for each region.

When given an ID different from the special ">" ID, XREADGROUP iterates simultaneously over all of the PELs for all consumers.
It returns the next entry by comparing entry IDs from the different PELs.

title: Conflict resolution

text:
The "delete wins" approach is a way to automatically resolve conflicts with consumer groups.
In case of concurrent consumer group operations, a delete will "win" over other concurrent operations on the same group.

In this example, the DEL at _t4_ deletes both the observed `group1` and the non-observed `group2`:

| Time | Region 1              | Region 2              |
| ---- | ----------------------- | ----------------------- |
| _t1_   | `XGROUP CREATE x group1 0`    |                         |
| _t2_   | _— Sync —_            | _— Sync —_            |
| _t3_   | `XINFO GROUPS x`
**→ [group1]** | `XINFO GROUPS x`
**→ [group1]** |
| _t4_   | `DEL x`                   | `XGROUP CREATE x group2 0`    |
| _t5_   | _— Sync —_            | _— Sync —_            |
| _t6_   | `EXISTS x`
**→ 0** | `EXISTS x`
**→  0**          |

In this example, the XGROUP DESTROY at _t4_ affects both the observed `group1` created in Region 1 and the non-observed `group1` created in Region 3:

| time | Region 1              | Region 2              | Region 3            |
| ---- | ----------------------- | ----------------------- | --------------------- |
| _t1_   | `XGROUP CREATE x group1 0`    |                         |                       |
| _t2_   | _— Sync —_            | _— Sync —_            |                       |
| _t3_   | `XINFO GROUPS x`
**→ [group1]** | `XINFO GROUPS x`
**→  [group1]** | `XINFO GROUPS x`
**→  []** |
| _t4_   |                         | `XGROUP DESTROY x group1`     | `XGROUP CREATE x group1 0` |
| _t5_   | _— Sync —_            | _— Sync —            | — Sync —          |
| _t6_   | `EXISTS x`
**→  0**          | `EXISTS x`
**→ 0**          | `EXISTS x`
**→  0**    |

title: Group replication

text:
Calls to XREADGROUP and XACK change the state of a consumer group or consumer. However, it's not efficient to replicate every change to a consumer or consumer group.

To maintain consumer groups in Active-Active databases with optimal performance:

1. Group existence (CREATE/DESTROY) is replicated.
1. Most XACK operations are replicated.
1. Other operations, such as XGROUP, SETID, DELCONSUMER, are not replicated.

For example:

| Time | Region 1                                        | Region 2               |
| ---- | ------------------------------------------------- | ------------------------ |
| _t1_   | `XADD messages 110 text hello`                                |                          |
| _t2_   | `XGROUP CREATE messages group1 0`                              |                          |
| _t3_   | `XREADGROUP GROUP group1 Alice STREAMS messages >`
**→ [110-1]** |                          |
| _t4_   | _— Sync —_                                      | _— Sync —_             |
| _t5_   | `XRANGE messages - +`
**→ [110-1]**                          | XRANGE messages - +
**→ [110-1]** |
| _t6_   | `XINFO GROUPS messages`
**→ [group1]**                            | XINFO GROUPS messages
**→ [group1]**   |
| _t7_   | `XINFO CONSUMERS messages group1`
**→ [Alice]**                      | XINFO CONSUMERS messages group1
**→ []**  |
| _t8_   | `XPENDING messages group1 - + 1`
**→ [110-1]**                     | XPENDING messages group1 - + 1
**→ []** |

Using XREADGROUP across regions can result in regions reading the same entries.
This is due to the fact that Active-Active Streams is designed for at-least-once reads or a single consumer.
As shown in the previous example, Region 2 is not aware of any consumer group activity, so redirecting the XREADGROUP traffic from Region 1 to Region 2 results in reading entries that have already been read.

title: Replication performance optimizations

text:
Consumers acknowledge messages using the XACK command. Each ack effectively records the last consumed message. This can result in a lot of cross-region traffic. To reduce this traffic, we replicate XACK messages only when all of the read entries are acknowledged.

| Time | Region 1                                                      | Region 2   | Explanation                                                                                |
| ---- | --------------------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------- |
| _t1_   | `XADD x 110-0 f1 v1`                                              |              |                                                                                |
| _t2_   | `XADD x 120-0 f1 v1`                                              |              |                                                                                |
| _t3_   | `XADD x 130-0 f1 v1`                                              |              |                                                                                |
| _t4_   | `XGROUP CREATE x group1 0`                                            |              |                                                                                |
| _t5_   | `XREADGROUP GROUP group1 Alice STREAMS x >` <br/>**→ [110-0, 120-0, 130-0]** |              |                                                                                |
| _t6_   | `XACK x group1 110-0`                                                   |              |                                                                                |
| _t7_   | _— Sync —_                                                    | _— Sync —_ | 110-0 and its preceding entries (none) were acknowledged. We replicate an XACK effect for 110-0.                |
| _t8_   | `XACK x group1 130-0`                                                   |              |                                                                                |
| _t9_   | _— Sync —_                                                    | _— Sync —_ | 130-0 was acknowledged, but not its preceding entries (120-0). We DO NOT replicate an XACK effect for 130-0     |
| _t10_  | `XACK x group1 120-0`                                                   |              |                                                                                |
| _t11_  | _— Sync —_                                                    | _— Sync —_ | 120-0 and its preceding entries (110-0 through 130-0) were acknowledged. We replicate an XACK effect for 130-0. |

In this scenario, if we redirect the XREADGROUP traffic from Region 1 to Region 2 we do not re-read entries 110-0, 120-0 and 130-0.
This means that the XREADGROUP does not return already-acknowledged entries.

title: Guarantees

text:
Unlike XREAD, XREADGOUP will never skip stream entries.

In traffic redirection, XREADGROUP may return entries that have been read but not acknowledged. It may also even return entries that have already been acknowledged.

When entries are added from multiple regions, Redis guarantees ordered entries in the reply of a single read command (`XREAD`, `XREADGROUP`, or `XRANGE`). However, ordered IDs are not guaranteed for a sequence of read commands.
For example, two subsequent `XREADGROUP GROUP group consumer COUNT 1 STREAMS key >` commands may reply with decreasing IDs.

title: Summary

text: With Active-Active streams, you can write to the same logical stream from multiple regions. As a result, the behavior of Active-Active streams differs somewhat from the behavior you get with Redis Open Source. This is summarized below:

title: Stream commands

text:
1. When using the _strict_ ID generation mode, XADD does not permit full stream entry IDs (that is, an ID containing both MS and SEQ).
1. XREAD may skip entries when iterating a stream that is concurrently written to from more than one region. For reliable stream iteration, use XREADGROUP instead.
1. XSETID fails when the new ID is less than current ID.

title: Consumer group notes

text:
The following consumer group operations are replicated:

1. Consecutive XACK operations
1. Consumer group creation and deletion (that is, XGROUP CREATE and XGROUP DESTROY)

All other consumer group metadata is not replicated.

A few other notes:

1. XGROUP SETID and DELCONSUMER are not replicated.
1. Consumers exist locally (XREADGROUP creates a consumer implicitly).
1. Renaming a stream (using RENAME) deletes all consumer group information.

code: rladmin tune db crdb crdt_xadd_id_uniqueness_mode strict

section_id: id-generation-modes

code: rladmin tune db crdb crdt_xadd_id_uniqueness_mode semi-strict

code: rladmin tune db crdb crdt_xadd_id_uniqueness_mode liberal


title: Strings and bitfields in Active-Active databases

summary: Information about using strings and bitfields with an Active-Active database.

title: Overview

text:
Active-Active databases support both strings and bitfields.

Active-Active **bitfield** support was added in RS version 6.0.20.

Changes to both of these data structures will be replicated across Active-Active member databases.

title: Replication semantics

text: Except in the case of [string counters]() (see below), both strings and bitfields are replicated using a "last write wins" approach. The reason for this is that strings and bitfields are effectively binary objects. So, unlike with lists, sets, and hashes, the conflict resolution semantics of a given operation on a string or bitfield are undefined.

title: How "last write wins" works

text:
A wall-clock timestamp (OS time) is stored in the metadata of every string
and bitfield operation. If the replication syncer cannot determine the order of operations,
the value with the latest timestamp wins. This is the only case with Active-Active databases where OS time is used to resolve a conflict.

Here's an example where an update happening to the same key at a later
time (t2) wins over the update at t1.

|  **Time** | **Region 1** | **Region 2** |
|  :------: | :------: | :------: |
|  t1 | SET text "a" |  |
|  t2 |  | SET text "b" |
|  t3 | — Sync — | — Sync — |
|  t4 | SET text "c" |  |
|  t5 | — Sync — | — Sync — |
|  t6 |  | SET text "d" |

title: String counter support

text:
When you're using a string as counter (for instance, with the [INCR]() or [INCRBY]() commands),
then conflicts will be resolved semantically.

On conflicting writes, counters accumulate the total counter operations
across all member Active-Active databases in each sync.

Here's an example of how counter
values works when synced between two member Active-Active databases. With
each sync, the counter value accumulates the private increment and
decrements of each site and maintain an accurate counter across
concurrent writes.

|  **Time** | **Region 1** | **Region 2** |
|  :------: | :------: | :------: |
|  t1 | INCRBY counter 7 |  |
|  t2 |  | INCRBY counter 3 |
|  t3 | GET counter
7 | GET counter
3 |
|  t4 | — Sync — | — Sync — |
|  t5 | GET counter
10 | GET counter
10 |
|  t6 | DECRBY counter 3 |  |
|  t7 |  | INCRBY counter 6 |
|  t8 | — Sync — | — Sync — |
|  t9 | GET counter
13 | GET counter
13 |

Active-Active databases support 59-bit counters.
This limitation is to protect from overflowing a counter in a concurrent operation.


title: Develop applications with Active-Active databases

summary: Overview of how developing applications differs for Active-Active databases from standalone Redis databases.

title: Overview

text:
Developing geo-distributed, multi-master applications can be difficult.
Application developers may have to understand a large number of race
conditions between updates to various sites, network, and cluster
failures that could reorder the events and change the outcome of the
updates performed across geo-distributed writes.

Active-Active databases (formerly known as CRDB) are geo-distributed databases that span multiple Redis Software clusters.
Active-Active databases depend on [multi-master replication](https://en.wikipedia.org/wiki/Multi-master_replication) and [conflict-free
replicated data types (CRDTs)](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) to power a simple development experience
for geo-distributed applications. Active-Active databases allow developers to use existing
Redis data types and commands, but automatically handle conflicting concurrent writes to the same key
across multiple geographies. For example, developers can simply use the
`INCR` or `INCRBY` method in Redis in all instances of the geo-distributed
application, and Active-Active databases handle the additive nature of `INCR` to reflect the
correct final value.

The following example displays a sequence of events
over time: t1 to t9. This Active-Active database has two member Active-Active databases: member CRDB1 and
member CRDB2. The local operations running in each member Active-Active database are
listed under the member Active-Active database name. The sync events represent the moment
when synchronization catches up to distribute all local member Active-Active database
updates to other participating clusters and other member Active-Active databases.

|  **Time** | **Member CRDB1** | **Member CRDB2** |
|  :------: | :------: | :------: |
|  t1 | INCRBY key1 7 |  |
|  t2 |  | INCRBY key1 3 |
|  t3 | GET key1
7 | GET key1
3 |
|  t4 | — Sync — | — Sync — |
|  t5 | GET key1
10 | GET key1
10 |
|  t6 | DECRBY key1 3 |  |
|  t7 |  | INCRBY key1 6 |
|  t8 | — Sync — | — Sync — |
|  t9 | GET key1
13 | GET key1
13 |

Databases provide various approaches to address some of these concerns:

- **Active-Passive geo-distributed deployments**: With active-passive
    distributions, all writes go to an active cluster. Redis Sofware
    provides a [Replica Of]() capability that provides a similar approach.
    This can be employed when the workload is heavily balanced toward
    reads and few writes. However, WAN performance and availability
    can be unreliable, and traveling large distances for writes takes away
    from application performance and availability.
- **Two-phase commit (2PC)**: This approach is designed around a protocol
    that commits a transaction across multiple transaction managers.
    Two-phase commit provides a consistent transactional write across
    regions but fails transactions unless all participating transaction
    managers are available at the time of the transaction. The number
    of messages exchanged and its cross-regional availability
    requirement make two-phase commit unsuitable for even moderate
    throughputs and cross-geo writes that go over WANs.
- **Sync update with quorum-based writes**: This approach synchronously
    coordinates a write across the majority of replicas across
    clusters spanning multiple regions. However, just like two-phase
    commit, the number of messages exchanged and its cross-regional
    availability requirement make geo-distributed quorum writes
    unsuitable for moderate throughputs and cross-geo writes that go
    over WANs.
- **Last-Writer-Wins (LWW) conflict resolution**: Some systems provide
    simplistic conflict resolution for all types of writes where
    system clocks are used to determine the winner across conflicting
    writes. LWW is lightweight and can be suitable for simpler data.
    However, LWW can be destructive to updates that are not necessarily
    conflicting. For example, adding a new element to a set across two
    geographies concurrently would result in only one of these new
    elements appearing in the final result with LWW.
- **MVCC (multi-version concurrency control)**: MVCC systems maintain
    multiple versions of data and may expose ways for applications to
    resolve conflicts. Even though an MVCC system can provide a flexible
    way to resolve conflicting writes, it comes at the cost of great
    complexity in the development of a solution.

Even though types and commands in Active-Active databases look identical to standard Redis
types and commands, the underlying types in Redis Software are enhanced to maintain
more metadata to create the conflict-free data type experience. This
section explains what you need to know about developing with Active-Active databases on
Redis Software.

title: Lua scripts

text:
Active-Active databases support Lua scripts, but unlike standard Redis, Lua scripts always
execute in effects replication mode. There is currently no way to
execute them in script-replication mode.

title: Eviction

text:
The default policy for Active-Active databases is _noeviction_ mode. Redis Software version 6.0.20 and later support all eviction policies for Active-Active databases, unless [Redis Flex or Auto Tiering]() (previously known as Redis on Flash) is enabled.

For details, see [eviction for Active-Active databases (Redis Software)]() or [eviction for Active-Active databases (Redis Cloud)]().

title: Expiration

text:
Expiration is supported with special multi-master semantics.

If a key's expiration time is changed at the same time on different
members of the Active-Active database, the longer extended time set via TTL on a key is
preserved.

If this command was performed on key1 on cluster #1:

[code example]

If this command was performed on key1 on cluster #2:

[code example]

The `EXPIRE` command setting the key to 50 would win.

And if this command was performed on key1 on cluster #3:

[code example]

It would win out of the three clusters hosting the Active-Active database as it sets the
TTL on key1 to an infinite time.

The replica responsible for the winning expire value is also
responsible for expiring the key and propagating a DEL effect when this
happens. From this point on, a losing replica is not responsible
for expiring the key, unless another `EXPIRE` command resets the TTL.
Furthermore, a replica that is not the owner of the expired value:

- Silently ignores the key if a user attempts to access it in READ
    mode, for example, treating it as if it was expired but not propagating a
    DEL.
- Expires it (sending a DEL) before making any modifications if a user
    attempts to access it in WRITE mode.

Expiration values are in the range of [0, 2^49] for Active-Active databases and [0, 2^64] for regular databases.

title: Tombstones

text:
For conflict resolution purposes, Active-Active databases cannot immediately release a deleted key. Instead, the key is logically deleted but remains in memory as a tombstone until the garbage collector can safely remove it.

When a deleted key becomes a tombstone, it frees some memory previously consumed by the key. The size of each tombstone varies depending on the data type and the key's history.

The garbage collector automatically removes a tombstone when all instances in the Active-Active database have observed the deletion operation.

To monitor tombstones, you can use shard-level metrics exposed by [`INFO crdt`](#info) and Grafana.

title: Out-of-Memory (OOM) {#outofmemory-oom}

text:
If a member Active-Active database is out of memory, that member is marked as
"inconsistent", the member stops responding to user traffic, and
the syncer initiates full reconciliation with other peers in the Active-Active database.

title: Active-Active database key counts

text:
Keys are counted differently for Active-Active databases:

- DBSIZE (in `shard-cli dbsize`) reports key header instances
    that represent multiple potential values of a key before a replication conflict is resolved.
- expired_keys (in `bdb-cli info`) can be more than the keys count in DBSIZE (in `shard-cli dbsize`)
    because expires are not always removed when a key becomes a tombstone.
- The Expires average TTL (in `bdb-cli info`) is computed for local expires only.

title: INFO

text:
The `INFO` command has an additional CRDT section that provides advanced troubleshooting information:

|  **Section** | **Field** | **Description** |
|  ------ | ------ | ------ |
|  **CRDT Context** | crdt_config_version | Current Active-Active database configuration version. |
|   | crdt_slots | Hash slots assigned and reported by this shard. |
|   | crdt_replid | Unique replica/shard IDs. |
|   | crdt_clock | Clock value of local vector clock. |
|   | crdt_ovc | Locally observed Active-Active database vector clock. |
|  **Peers** | | A list of currently connected peer replication peers. This is similar to the replicas list reported by Redis. |
|  **Backlogs** | | A list of peer replication backlogs currently maintained. Typically in a full mesh topology, only a single backlog is used for all peers, as the requested IDs are identical. |
|  **CRDT Stats** | crdt_sync_full | Number of inbound full synchronization processes performed. |
|   | crdt_sync_partial_ok | Number of partial (backlog based) re-synchronization processes performed. |
|   | crdt_sync_partial-err | Number of partial re-synchronization processes failed due to exhausted backlog. |
|   | crdt_merge_reqs | Number of inbound merge requests processed. |
|   | crdt_effect_reqs | Number of inbound effect requests processed. |
|   | crdt_ovc_filtered_effect_reqs | Number of inbound effect requests filtered due to old vector clock. |
|   | crdt_gc_pending | Number of elements pending garbage collection. |
|   | crdt_gc_attempted | Number of attempts to garbage collect tombstones. |
|   | crdt_gc_collected | Number of tombstones garbage collected successfully. |
|   | crdt_gc_gvc_min | The minimal globally observed vector clock, as computed locally from all received observed clocks. |
|   | crdt_stale_released_with_merge | Indicates the last stale flag transition was a result of a complete full sync. |
|  **CRDT Replicas** | | A list of crdt_replica \<uid> entries, each describes the known state of a remote instance with the following fields: |
|   | config_version | Last configuration version reported. |
|   | shards | Number of shards. |
|   | slots | Total number of hash slots. |
|   | slot_coverage | A flag indicating remote shards provide full coverage (all shards are alive). |
|   | max_ops_lag | Number of local operations not yet observed by the least updated remote shard. |
|   | min_ops_lag | Number of local operations not yet observed by the most updated remote shard. |

section_id: expiration


title: Active-Active Redis applications

summary: General information to keep in mind while developing applications for an Active-Active database.

summary: Introduction to differences in data types between standalone and Active-Active Redis databases.

title: Data types for Active-Active databases

summary: How to failover your application to connect to a remote replica.

title: Application failover with Active-Active databases

title: Overview

text:
Developing globally distributed applications can be challenging, as
developers have to think about race conditions and complex combinations
of events under geo-failovers and cross-region write conflicts. In Redis Software (RS), Active-Active databases
simplify developing such applications by directly using built-in smarts
for handling conflicting writes based on the data type in use. Instead
of depending on just simplistic "last-writer-wins" type conflict
resolution, geo-distributed Active-Active databases (formerly known as CRDBs) combine techniques defined in CRDT
(conflict-free replicated data types) research with Redis types to
provide smart and automatic conflict resolution based on the data type's
intent.

An Active-Active database is a globally distributed database that spans multiple Redis
Enterprise Software clusters. Each Active-Active database can have many Active-Active database instances
that come with added smarts for handling globally distributed writes
using the proven
[CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
approach.
[CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
research describes a set of techniques for creating systems that can
handle conflicting writes. CRDBs powered by Multi-Master Replication
(MMR) provide a straightforward and effective way to replicate your
data between regions and simplify development of complex applications
that can maintain correctness under geo-failovers and concurrent
cross-region writes to the same data.

Active-Active databases replicate data between multiple Redis Software
clusters. Common uses for Active-Active databases include disaster recovery,
geographically redundant applications, and keeping data closer to your
users' locations. MMR is always multi-directional amongst the clusters
configured in the Active-Active database. For unidirectional replication, please see the
Replica Of capabilities in Redis Software.

title: Example of synchronization

role: example

text:
[Learn more about
synchronization for
each supported data type]() and [how to develop applications]() with them on Redis Software.


title: Application-based disaster recovery

summary: Application-based disaster recovery for Active-Active databases using custom application-level monitoring and connectivity management.

title: Content

text:
For complete control over failover and failback, you can implement disaster recovery mechanisms directly in the application server.

For more information, see [Application failover with Active-Active databases]().


title: Client library-based disaster recovery

summary: Client library-based disaster recovery for Active-Active databases using Redis client libraries with built-in failover logic.

title: Content

text:
Some Redis client libraries support geographic failover and failback. These client libraries monitor all Active-Active database members and instantiate connections for all endpoints in advance to allow faster failover and failback.

Advantages:

- No additional hardware or software components required.

- No high availability considerations.

- No scalability concerns.

- Tighter control over connectivity, such as timeouts, connection retries, and dynamic reconfiguration.

- OSS Cluster API support (currently only available in redis-py)

- Low latency.

Considerations:

- Requires code changes for failover and failback logic.

- Concurrent access across replicas is possible, but can be mitigated using the distributed health status provided by the database availability API requests.

- When a development framework uses Redis transparently, failover and failback might not be easy to configure.

The following diagram shows a client library-based disaster recovery approach:

The following diagram shows a client-based disaster recovery approach that also uses [connection pooling]():

For additional information, see the following client library guides for failover and failback:

- [Jedis (Java)]()

- [redis-py (Python)]()


summary: Disaster recovery strategies for Active-Active databases using network, proxy, client library, and application-based approaches.

summary: Network-based disaster recovery for Active-Active databases using global traffic managers and load balancing solutions.

title: Network-based disaster recovery

summary: Proxy-based disaster recovery for Active-Active databases.

title: Proxy-based disaster recovery

title: Client library-based disaster recovery

title: Application-based disaster recovery

title: Overview

text:
An application deployed with an Active-Active database connects to a database member that is geographically nearby. If that database member becomes unavailable, the application can fail over to a secondary Active-Active database member, and fail back to the original database member again if it recovers.

However, because Active-Active Redis databases do not have a built-in [failover](https://en.wikipedia.org/wiki/Failover) or failback mechanism for application connections, you must implement one of the following [disaster recovery strategies](#disaster-recovery-strategies).

title: Disaster recovery strategies

text:
Depending on your requirements for Recovery Point Objective, Recovery Time Objective, consistency, scalability, resources, maintainability, and other factors, choose one of the following strategies to fail over to a secondary Active-Active member or fail back to the primary member:

- [Network-based](): Global traffic managers and load balancers handle routing at the network layer, requiring no application changes.

- [Proxy-based](): Software proxies handle detection and routing logic.

- [Client library-based](): Redis client libraries with built-in failover logic.

- [Application-based](): Custom application-level monitoring and connectivity management.

title: Considerations for disaster recovery

text:
When implementing a disaster recovery strategy for an Active-Active database, consider the following:

- Is the Active-Active database an on-premise, cloud, multi-cloud, or hybrid-cloud deployment?

- Number of regions and availability zones.

- Application server redundancy and deployment locations.

- Acceptable values for the maximum amount of data that can be lost during a failure (Recovery Point Objective) and the maximum acceptable time to restore service after a failure (Recovery Time Objective).

- Latency and throughput requirements.

- Number of application errors that can be tolerated during a failure.

- Tolerance for reading stale but eventually consistent data during a failover scenario.

- Is concurrent access, in which different application servers can read from or write to different Active-Active database members, acceptable?

- Are there any regulatory or policy requirements for disaster recovery?

- Does the application connect to the Active-Active database using a Redis client library or through a development framework or ecosystem?

- Does the Active-Active database use DNS, the [OSS Cluster API](), or the [discovery service]()?

- Is rate-limiting control needed?

- Can you modify the existing codebase or introduce new components, such as load balancers or proxies?

title: Detect failures with health checks

text:
Depending on the failover strategy, you can use the following health checks to detect Active-Active database failures and determine whether to fail over to a secondary Active-Active member, or fail back to the primary member after the preferred endpoint is back online.

To determine which health checks to use, consider factors such as detection speed, integrability with the failover strategy, and writability or durability guarantees.

text:
Lag-aware database availability requests are the recommended method to detect database failures in Redis Software deployments. This method guarantees that all the shards of a clustered database are connectable.

See [Lag-aware database availability requests]() for more information.

Lag-aware database availability requests are not supported for Redis Cloud databases.

title: Redis connection health checks

text:
You can use an existing connection to the database to check its availability.

#### PING command

The [`PING`]() command checks the following:

- The database is connectable.

- The database is readable.

- The dataset is available.

Example response for an available database:

[code example]

If a database is connectable but not available for reads, such as when reloading from a snapshot, `PING` returns an error message:

[code example]

#### Connection timeouts or Redis errors

By capturing connection errors, you can determine when to fail over to a secondary Active-Active member or fail back to the primary member based on the [circuit breaker pattern](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern).

#### Custom health check

You can also implement custom health checks.

For example, you can check the keyspace with write operations such as using the [`SET`]() command to write arbitrary data. This check verifies that database shards are available and writable.

For example:

[code example]

Use multiple write operations with different randomized keys to access different shards and guarantee that all the shards are available.

title: Health check comparison

text:
| Health check | Connectivity | Readability | Writability | Durability | Notes |
|--------------|--------------|-------------|--------------|------------|-------|
| Database availability requests |:white_check_mark: |  |  |  | No guarantees on readability. For example, the shard might be reloading from a snapshot. |
| `PING` |:white_check_mark: |:white_check_mark: |:white_check_mark: |  | No support for clustered databases. All `PING` requests will be forwarded to shard 1. |
| Keyspace sampling |:white_check_mark: |:white_check_mark: |:white_check_mark: |:white_check_mark: | Write operations are persisted, increasing disk usage for AOF and RDB. |

section_id: redis-connection-health-checks

code:
127.0.0.1:6379> PING
(error) LOADING Redis is loading the dataset in memory

code: SET <randomized_key_name> <arbitrary_value> EX 1


title: Network-based disaster recovery

title: Overview

text:
For cross-region availability, you can use a global traffic manager or a global load balancer. Both help route traffic across regions, but they work in different ways:

- Global traffic managers are DNS-based and don't handle traffic directly. Instead, they answer DNS queries with the best IP address to connect to, based on factors such as distance, latency, or health. After DNS resolution, the client connects directly to the endpoint.

- Global load balancers can route actual traffic and make decisions in real time, using request headers or connection state.

If your deployment does not require cross-region availability, you can use a regional load balancer to grant cross-zone redundancy.

title: Cross-region availability

text:
For cross-region availability, you can use a global traffic manager or a global load balancer.

Advantages:

- If DNS routing is available at the application level, no additional load balancer is required between the application and the data tier to resolve the Active-Active database member's FQDN, reducing latency.

- Protects against data center failure since failure in one region should not affect services running in another region.

title: Global traffic manager

text:
A global traffic manager acts as an intelligent DNS server that directs clients to healthy endpoints based on distance, latency, or availability. You should configure the traffic manager to route to the local region first and fail over to other regions if an issue occurs.

Advantages:

- High availability.

- Latency optimization.

- Seamless disaster recovery.

Considerations:

- DNS propagation delays affect failover time.

- DNS caches can impact proper functioning.

- Limited custom health check support.

- May route traffic during CRDT synchronization, causing stale data reads.

The following diagram shows how a global traffic manager with DNS resolution routes traffic:

If the environment does not allow DNS resolution, you can use a load balancer to direct traffic to the cluster nodes:

title: Global load balancer

text:
For real-time traffic control and more advanced routing logic for cross-region failover and failback, you can use a global load balancer. However, this solution can have higher latency than a global traffic manager.

The following diagram shows how a global load balancer routes traffic between regions:

title: Cross-zone availability

text:
If your deployment does not require cross-region availability, you can use a regional load balancer to route requests to a healthy Active-Active database member in a different availability zone within the same region.

The following diagram shows how a regional load balancer routes traffic across availability zones:

title: Advantages

text:
Using a global traffic manager or global load balancer for disaster recovery has the following advantages:

- Because the routing logic happens at the network level:

    - No code changes are required. For applications, there is less logic to manage, and fewer working parts that can break.

    - Development frameworks are agnostic and can connect to a single member Active-Active database endpoint.

- If DNS routing is available at the application level, no additional load balancer is required between the application and the data tier to resolve the member Active-Active database’s FQDN, reducing latency.

- Failure of any kind in one region should not affect services running in another. This approach is ideal for scenarios where an entire data center is compromised, and service continuity is granted at a data center level.

title: Considerations

text:
If you plan to implement network-based disaster recovery, you should also consider the following challenges:

- Standard health check mechanisms are available, but custom health checks may not be supported. Proper availability mechanisms, such as the database availability request, must be configured directly where possible or provided as an additional lightweight service alongside the application.

- Global load balancers introduce latency.

- Refreshing DNS records takes some time to propagate, which challenges the ability to control the failover time. In addition, DNS caches, often used by operating systems or the JVM, can impact the proper functioning of this approach.

- When failback is required, routing traffic from a secondary Active-Active database member to the primary database member can happen during CRDT synchronization. If the failback's target has not completed synchronization, applications might read stale data.

- The Active-Active health check policy should be designed to use the lag-aware database availability request, thus granting control over the observed CRDT replication lag. This method increases consistency in a failback scenario. However, specific global traffic managers and global load balancers might require extra effort to set the Active-Active health check policy.

- Replicating the application stack for several regions impacts planning, maintenance, and costs.


title: Proxy-based disaster recovery

title: Overview

text:
If you add a lightweight proxy software component between the clients and the Active-Active database, applications can dynamically route requests to the optimal endpoint.

Advantages:

- Proxies provide proactive and reactive health check methods, such as polling target health periodically using either a TCP connection or an HTTP request, or monitoring live operations for errors.

- Proxies can be configured to run Active-Active health checks, such as the lag-aware database availability requests.

- If an Active-Active database member fails, a proxy can automatically detect the issue and redirect traffic to a healthy Active-Active database member without requiring DNS propagation delays or client disconnections. This enables fast, controlled failover and minimizes downtime.

Considerations:

- If you do not use DNS to resolve the Active-Active database members' FQDNs:

    - The proxies must have static IPs.

    - If you add a new node to the cluster, you must configure the proxy with the new endpoint.

    - A configuration syncer component is required to discover topology changes and reconfigure the proxy.

- Proxies introduce latency.

- Proxy failures can disconnect clients and cause disruptions.

title: Avoid concurrent access across replicas

text:
If concurrent access across replicas must be avoided in every scenario, you can use a centralized proxy with a standby proxy instance for high availability.

Advantages:

- Prevents concurrent access across replicas.

- Failover and failback are simultaneous regardless of the Active-Active health check policy.

Considerations:

- Although the proxy can be monitored with a watchdog and restarted in case of failure, this setup does not grant high availability for the proxy.

- Limited scalability.

The following diagram shows a centralized proxy architecture with a standby proxy instance:

title: Co-locate to reduce latency and improve scalability

text:
To reduce latency and improve scalability, you can use a proxy co-located in the application server.

Advantages:

- Reduced latency.

- Better scalability.

Considerations:

- Failover and failback might not be simultaneous depending on the Active-Active health check policy.

The following diagram shows a co-located proxy architecture where each application server has its own proxy:

title: Pool proxies for scalability

text:
You can use a pool of active proxies to scale the routing layer. Application servers can balance new connections to the pool of proxies using a round-robin distribution algorithm, such as DNS-based round robin.

Advantages:

- High availability without complex monitoring and failover solutions.

- Flexible scalability of the routing layer.

Considerations:

- Concurrent access across replicas is possible, but can be mitigated using database availability API requests.

The following diagram shows a pool of proxies:


title: Get started with Redis Software Active-Active databases

summary: Quick start guide to create an Active-Active database for test and development.

title: Overview

text:
To get started, this article will help you set up an Active-Active database, formerly known as CRDB (conflict-free replicated database), spanning across two Redis Software
clusters for test and development environments. Here are the steps:

1. Run two Redis Software Docker containers.

1. Set up each container as a cluster.

1. Create a new Redis Software Active-Active database.

1. Test connectivity to the Active-Active database.

To run an Active-Active database on installations from the [Redis Software download package](),
set up two Redis Software installations and continue from Step 2.

This getting started guide is for development or demonstration environments.
For production environments, see [Create an Active-Active geo-replicated database]() for instructions.

title: Run two containers

text:
To spin up two Redis Software containers, run these commands:

[code example]

[code example]

The `-h` option sets the hostname of the container, which is important for cluster setup and identification.

The `--name` option assigns a name to the container, making it easier to manage and reference with Docker commands.

The `-p` options map the Cluster Manager UI port (8443), REST API port (9443), and database access port differently for each container to make sure that all containers can be accessed from the host OS that is running the containers.

title: Set up two clusters

text:
1. For cluster 1, go to `https://localhost:8443` in a browser on the
host machine to access the Redis Software Cluster Manager UI.

Depending on your browser, you may see a certificate error. Continue to the website.

1. Click **Create new cluster**:

1. Enter an email and password for the administrator account, then click **Next** to proceed to cluster setup:

1. Enter your cluster license key if you have one. Otherwise, a trial version is installed.

1. In the **Configuration** section of the **Cluster** settings page, enter a cluster FQDN, for example `cluster1.local`:

1. On the node setup screen, keep the default settings and click **Create cluster**:

1. Click **OK** to confirm that you are aware of the replacement of the HTTPS SSL/TLS
    certificate on the node, and proceed through the browser warning.

1. Repeat the previous steps for cluster 2 with these differences:

    - In your web browser, go to `https://localhost:8445` to set up the cluster 2.

    - For the **Cluster name (FQDN)**, enter a different name, such as `cluster2.local`.

Now you have two Redis Software clusters with FQDNs
`cluster1.local` and `cluster2.local`.

Each Active-Active instance must have a unique fully-qualified domain name (FQDN).

title: Create an Active-Active database

text:
1. Sign in to cluster1.local's Cluster Manager UI at `https://localhost:8443`

1. Open the **Create database** menu with one of the following methods:

    - Click the **+** button next to **Databases** in the navigation menu:

    - Go to the **Databases** screen and select **Create database**:

1. Select **Active-Active database**.

1. Enter the cluster's local admin credentials, then click **Save**:

1. Add participating clusters that will host instances of the Active-Active database:

    1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

    1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

        In the **Other participating clusters** list, add the address and admin credentials for the other cluster: `https://cluster2.local:9443`

    1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Enter `database1` for **Database name** and `12000` for **Port**:

1. Configure additional settings:

    1. In the **High availability** section, turn off **Replication** since each cluster has only one node in this setup:

    1. In the **Clustering** section, either:

        - Make sure that **Sharding** is enabled and select the number of shards you want to have in the database. When database clustering is enabled,
        databases are subject to limitations on [Multi-key commands]().
        You can increase the number of shards in the database at any time.

        - Turn off **Sharding** to use only one shard and avoid [Multi-key command]() limitations.

You cannot enable or turn off database clustering after the Active-Active database is created.

1. Click **Create**.

1. After the Active-Active database is created, sign in to the Cluster Manager UIs for cluster 1 at `https://localhost:8443` and cluster 2 at `https://localhost:8445`.

1. Make sure each cluster has an Active-Active database member database with the name `database1`.

    In a real-world deployment, cluster 1 and cluster 2 would most likely be
    in separate data centers in different regions. However, for
    local testing we created the scale-minimized deployment using two
    local clusters running on the same host.

<!-- Also in getting-started-crdbs.md -->

title: Test connection

text:
With the Redis database created, you are ready to connect to your
database. See [Connect to Active-Active databases]() for tutorials and examples of multiple connection methods.

code: docker run -d --cap-add sys_resource -h rs1_node1 --name rs1_node1 -p 8443:8443 -p 9443:9443 -p 12000:12000 redislabs/redis

section_id: run-two-containers

code: docker run -d --cap-add sys_resource -h rs2_node1 --name rs2_node1 -p 8445:8443 -p 9445:9443 -p 12002:12000 redislabs/redis


title: Active-Active geo-distributed Redis

summary: Overview of the Active-Active database in Redis Software

summary: Information about Active-Active database to take into consideration while planning a deployment, such as compatibility, limitations, and special configuration

title: Considerations for planning Active-Active databases

summary: How to create an Active-Active database and things to consider when setting it up.

title: Create an Active-Active geo-replicated database

summary: How to connect to an Active-Active database using redis-cli or a sample Python application.

title: Connect to your Active-Active databases

summary: Manage your Active-Active database settings.

title: Manage Active-Active databases

summary: Considerations while deleting Active-Active databases.

title: Delete Active-Active databases

title: Disaster recovery strategies for Active-Active databases

summary: Enable causal consistency in an Active-Active database.

title: Enable causal consistency

summary: How to configure distributed synchronization so that any available proxy endpoint can manage synchronization traffic.

title: Configure distributed synchronization

summary: Detailed information about the syncer process and its role in distributed databases.

title: Syncer process

title: Active-Active Redis applications

title: Overview

text:
In Redis Software, Active-Active geo-distribution is based on [CRDT technology](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type).
The Redis Software implementation of CRDT is called an Active-Active database (formerly known as CRDB).
With Active-Active databases, applications can read and write to the same data set from different geographical locations seamlessly and with latency less than one millisecond (ms),
without changing the way the application connects to the database.

Active-Active databases also provide disaster recovery and accelerated data read-access for geographically distributed users.

title: High availability

text:
The [high availability]() that Active-Active replication provides is built upon a number of Redis Software features (such as [clustering](), [replication](), and [replica HA]()) as well as some features unique to Active-Active ([multi-primary replication](), [automatic conflict resolution](), and [strong eventual consistency]()).

Clustering and replication are used together in Active-Active databases to distribute multiple copies of the dataset across multiple nodes and multiple clusters. As a result, a node or cluster is less likely to become a single point of failure. If a primary node or primary shard fails, a replica is automatically promoted to primary. To avoid having one node hold all copies of certain data, the [replica HA]() feature (enabled by default) automatically migrates replica shards to available nodes.

title: Multi-primary replication

text:
In Redis Software, replication copies data from primary shards to replica shards. Active-Active geo-distributed replication also copies both primary and replica shards to other clusters. Each Active-Active database needs to span at least two clusters; these are called participating clusters.

Each participating cluster hosts an instance of your database, and each instance has its own primary node. Having multiple primary nodes means you can connect to the proxy in any of your participating clusters. Connecting to the closest cluster geographically enables near-local latency. Multi-primary replication (previously referred to as multi-master replication) also means that your users still have access to the database if one of the participating clusters fails.

Active-Active databases do not replicate the entire database, only the data.
Database configurations, LUA scripts, and other support info are not replicated.

title: Syncer

text:
Keeping multiple copies of the dataset consistent across multiple clusters is no small task. To achieve consistency between participating clusters, Redis Active-Active replication uses a process called the [syncer]().

The syncer keeps a [replication backlog](), which stores changes to the dataset that the syncer sends to other participating clusters. The syncer uses partial syncs to keep replicas up to date with changes, or a full sync in the event a replica or primary is lost.

title: Conflict resolution

text:
Because you can connect to any participating cluster to perform a write operation, concurrent and conflicting writes are always possible. Conflict resolution is an important part of the Active-Active technology. Active-Active databases only use [conflict-free replicated data types (CRDTs)](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). These data types provide a predictable conflict resolution and don't require any additional work from the application or client side.

When developing with CRDTs for Active-Active databases, you need to consider some important differences. See [Develop applications with Active-Active databases]() for related information.

title: Strong eventual consistency

text:
Maintaining strong consistency for replicated databases comes with tradeoffs in scalability and availability. Redis Active-Active databases use a strong eventual consistency model, which means that local values may differ across replicas for short periods of time, but they all eventually converge to one consistent state. Redis uses vector clocks and the CRDT conflict resolution to strengthen consistency between replicas. You can also enable the causal consistency feature to preserve the order of operations as they are synchronized among replicas.

Other Redis Software features can also be used to enhance the performance, scalability, or durability of your Active-Active database. These include [data persistence](), [multiple active proxies](), [distributed synchronization](), [OSS Cluster API](), and [rack-zone awareness]().

title: Next steps

text:
- [Plan your Active-Active deployment]()
- [Get started with Active-Active]()
- [Create an Active-Active database]()
- [Develop applications with Active-Active databases]()
- Review [disaster recovery strategies for Active-Active databases]()


title: Manage Active-Active databases

title: Overview

text: You can configure and manage your Active-Active database from either the Cluster Manager UI or the command line.

title: Database settings

text:
Most Active-Active database settings can be changed after database creation. One notable exception is database clustering, which can't be turned on or off after the database has been created.

As of Redis Software version 8.0.16, the Cluster Manager UI supports both [global](#change-global-configuration) and [local](#change-local-configuration) configuration changes for Active-Active databases. In earlier versions, configuration changes made in the Cluster Manager UI applied only to the local instance and required additional manual updates for each participating cluster.

title: Change global configuration

text:
Global configuration changes are applied to all participating clusters in the Active-Active database. However, if your Active-Active mesh contains local configurations, they will continue to override global settings on the database instances where they are applied.

Applying global configuration changes will override any local settings created using the API or earlier Cluster Manager UI versions. To keep local changes, manage them in the [**Local** configuration tab](#change-local-configuration).

**Cluster Manager UI:**

To make global configuration changes in the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. While on the **Global** tab, click **Edit**.

1. Make your configuration changes.

1. Click **Save** to apply the changes globally to all participating clusters.

**crdb-cli:**

To change the global configuration from the command line, use [`crdb-cli crdb update`]():

[code example]

Replace the placeholders `<placeholder>` with your own values.

title: Change local configuration

text:
Local configuration changes override global configuration settings and are applied only to the database instance on the current cluster.

To apply changes to a local Active-Active database instance, use one of the following methods:

**Cluster Manager UI:**

To change the local configuration in the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Go to the **Local** tab.

    <img src="../../../../../images/rs/screenshots/databases/active-active-databases/local-config-tab.png" alt="The local configuration tab is selected.">

1. Click **Edit**, then make your configuration changes.

1. Click **Save** to apply the changes to the current database instance only.

**rladmin:**

To change the local configuration from the command line, use [`rladmin tune db`]():

[code example]

Replace the placeholders `<placeholder>` with your own values.

title: View differences between global and local configuration

text:
In the Cluster Manager UI, an Active-Active database's **Global** configuration tab indicates when a local configuration differs from the global configuration:

If a warning symbol appears next to a setting on the **Global** configuration tab, point to the warning symbol to show details about the local configuration differences:

On the **Local** configuration tab, any locally configured settings that differ from the global settings are marked with a **Local configuration** label:

title: Participating clusters

text:
You can add and remove participating clusters of an Active-Active database to change the topology.
To manage the changes to Active-Active topology, use [`crdb-cli`]() or the participating clusters list in the Cluster Manager UI.

title: Add participating clusters

text:
All existing participating clusters must be online and in a syncing state when you add new participating clusters.

New participating clusters create the Active-Active database instance based on the global Active-Active database configuration.
After you add new participating clusters to an existing Active-Active database,
the new database instance can accept connections and read operations.
The new instance does not accept write operations until it is in the syncing state.

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

To add a new participating cluster to an existing Active-Active configuration using the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Click **Edit**.

1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Click **Save**.

title: Remove participating clusters

text:
All existing participating clusters must be online and in a syncing state when you remove an online participating cluster.
If you must remove offline participating clusters, you can forcefully remove them.
If a forcefully removed participating cluster tries to rejoin the cluster,
its Active-Active database membership will be out of date.
The joined participating clusters reject updates sent from the removed participating cluster.
To prevent rejoin attempts, purge the forcefully removed instance from the participating cluster.

To remove a participating cluster using the Cluster Manager UI:

1. Select the Active-Active database from the **Databases** list and go to its **Configuration** screen.

1. Click **Edit**.

1. In the **Participating clusters** section, point to the cluster you want to delete in the **Other participating clusters** list:

1. Click  to remove the cluster.

1. Click **Save**.

title: Replication backlog

text:
Redis databases that use [replication for high availability]() maintain a replication backlog (per shard) to synchronize the primary and replica shards of a database. In addition to the database replication backlog, Active-Active databases maintain a backlog (per shard) to synchronize the database instances between clusters.

By default, both the database and Active-Active replication backlogs are set to one percent (1%) of the database size divided by the number of shards. This can range between 1MB to 250MB per shard for each backlog.

title: Change the replication backlog size

text:
Use the [`crdb-cli`]() utility to control the size of the replication backlogs. You can set it to `auto` or set a specific size.

Update the database replication backlog configuration with the `crdb-cli` command shown below.

[code example]

Update the Active-Active (CRDT) replication backlog with the command shown below:

[code example]

title: Data persistence

text:
Active-Active supports AOF (Append-Only File) data persistence only.  Snapshot persistence is _not_ supported for Active-Active databases and should not be used.

If an Active-Active database is currently using snapshot data persistence, use `crdb-cli` to switch to AOF persistence:
[code example]

code: crdb-cli crdb update --crdb-guid <guid> --<setting-name> <setting-value>

section_id: change-global-configuration

code: rladmin tune db { db:<id> | <name> } <setting-name> <setting-value>

section_id: change-local-configuration

code: crdb-cli crdb update --crdb-guid <crdb_guid> --default-db-config "{\"repl_backlog_size\": <size in MB | 'auto'>}"

section_id: change-the-replication-backlog-size

code: crdb-cli crdb update --crdb-guid <crdb_guid> --default-db-config "{\"crdt_repl_backlog_size\": <size in MB | 'auto'>}"

code: crdb-cli crdb update --crdb-guid <CRDB_GUID> --default-db-config '{"data_persistence": "aof", "aof_policy":"appendfsync-every-sec"}'

section_id: data-persistence


title: Overview

text:
In Redis Software, Active-Active geo-distribution is based on [conflict-free replicated data type (CRDT) technology](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). Compared to databases without geo-distribution, Active-Active databases have more complex replication and networking, as well as a different data type.

Because of the complexities of Active-Active databases, there are special considerations to keep in mind while planning your Active-Active database.

See [Active-Active Redis]() for more information about geo-distributed replication. For more info on other high availability features, see [Durability and high availability]().

title: Participating clusters

text:
You need at least [two participating clusters]() for an Active-Active database. If your database requires more than ten participating clusters, contact Redis support. You can [add or remove participating clusters]() after database creation.

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

Changes made from the Cluster Manager UI to an Active-Active database configuration only apply to the cluster you are editing. For global configuration changes across all clusters, use the `crdb-cli` command-line utility.

title: Memory limits

text:
Database memory limits define the maximum size of your database across all database replicas and [shards]() on the cluster. Your memory limit also determines the number of shards.

Besides your dataset, the memory limit must also account for replication, Active-Active metadata, and module overhead. These features can increase your database size, sometimes increasing it by two times or more.

Factors to consider when sizing your database:

- **dataset size**: you want your limit to be above your dataset size to leave room for overhead.
- **database throughput**: high throughput needs more shards, leading to a higher memory limit.
- [**modules**](): using modules with your database can consume more memory.
- [**database clustering**](): enables you to spread your data into shards across multiple nodes (scale out).
- [**database replication**](): enabling replication doubles memory consumption
- [**Active-Active replication**](): enabling Active-Active replication requires double the memory of regular replication, which can be up to two times (2x) the original data size per instance.
- [**database replication backlog**]() for synchronization between shards. By default, this is set to 1% of the database size.
- [**Active-Active replication backlog**]() for synchronization between clusters. By default, this is set to 1% of the database size.

It's also important to know Active-Active databases have a lower threshold for activating the eviction policy, because it requires propagation to all participating clusters. The eviction policy starts to evict keys when one of the Active-Active instances reaches 80% of its memory limit.

For more information on memory limits, see [Memory and performance]() or [Database memory limits]().

title: Networking

text:
Network requirements for Active-Active databases include:

- A VPN between each network that hosts a cluster with an instance (if your database spans WAN).
- A network connection to [several ports](#network-ports) on each cluster from all nodes in all participating clusters.
- A [network time service](#network-time-service) running on each node in all clusters.

Networking between the clusters must be configured before creating an Active-Active database. The setup will fail if there is no connectivity between the clusters.

title: Network ports

text: Every node must have access to the REST API ports of every other node as well as other ports for proxies, VPNs, and the Cluster Manager UI. See [Network port configurations]() for more details. These ports should be allowed through firewalls that may be positioned between the clusters.

title: Network Time Service {#network-time-service}

text:
Active-Active databases require a time service like NTP or Chrony to make sure the clocks on all cluster nodes are synchronized.
This is critical to avoid problems with internal cluster communications that can impact your data integrity.

See [Synchronizing cluster node clocks]() for more information.

title: Redis modules {#redis-modules}

text:
Several Redis modules are compatible with Active-Active databases. Find the list of [compatible Redis modules]().

Active-Active databases created with or upgraded to Redis version 8 or later automatically enable [search and query]() and [JSON](), which allows you to index, query, and perform full-text searches of nested JSON documents.

title: Limitations

role: limits

text:
Active-Active databases have the following limitations:

- An existing database can't be changed into an Active-Active database. To move data from an existing database to an Active-Active database, you must [create a new Active-Active database]() and [migrate the data]().
- [Discovery service]() is not supported with Active-Active databases. Active-Active databases require FQDNs or [mDNS]().
- The `FLUSH` command is not supported from the CLI. To flush your database, use the API or Cluster Manager UI.
- The `UNLINK` command is a blocking command for all types of keys.
- Cross slot multi commands (such as `MSET`) are not supported with Active-Active databases.
- The hashing policy can't be changed after database creation.
- Active-Active databases cannot be configured as Redis Flex deployments.
- Active-Active databases handle replication internally and do not support the `redis.set_repl()` function in Lua scripts.
- If you enabled the default database password during the creation of an Active-Active database, you should not turn off the default database password because it could prevent the removal of participating database instances.


title: Syncer process

title: Syncer process

text:
Each node in a cluster containing an instance of an Active-Active database hosts a process called the syncer.
The syncer process:

1. Connects to the proxy on another participating cluster
1. Reads data from that database instance
1. Writes the data to the local cluster's primary(master) shard

Some replication capabilities are also included in [Redis Open Source]().

The primary (also known as master) shard at the top of the primary-replica tree creates a replication ID.
This replication ID is identical for all replicas in that tree.
When a new primary is appointed, the replication ID changes, but a partial sync from the previous ID is still possible.

In a partial sync, the backlog of operations since the offset are transferred as raw operations.
In a full sync, the data from the primary is transferred to the replica as an RDB file which is followed by a partial sync.

Partial synchronization requires a backlog large enough to store the data operations until connection is restored. See [replication backlog]() for more info on changing the replication backlog size.

title: Syncer in Active-Active replication

text:
In the case of an Active-Active database:

- Multiple past replication IDs and offsets are stored to allow for multiple syncs
- The [Active-Active replication backlog]() is also sent to the replica during a full sync.

Full sync triggers heavy data transfers between geo-replicated instances of an Active-Active database.

An Active-Active database uses partial synchronization in the following situations:

- Failover of primary shard to replica shard
- Restart or crash of replica shard that requires sync from primary
- Migrate replica shard to another node
- Migrate primary shard to another node as a replica using failover and replica migration
- Migrate primary shard and preserve roles using failover, replica migration, and second failover to return shard to primary

Synchronization of data from the primary shard to the replica shard is always a full synchronization.

title: Troubleshooting syncer errors

role: errors

title: Unrecoverable syncer errors

text:
Some syncer errors are unrecoverable and cause the syncer to exit with exit code 4. When this occurs, the Data Management Controller (DMC) automatically sets the `crdt_sync` or `replica_sync` value to `stopped`.

#### Restart syncer for regular databases

To restart a regular database's syncer after an unrecoverable error, [update the database configuration]() with the REST API to enable `sync`:

[code example]

#### Restart syncer for Active-Active databases

To restart an Active-Active database's syncer after an unrecoverable error, use one of the following methods.

-  For each participating cluster, [update the database configuration]() with the REST API to enable `sync`:

   [code example]

- Run [`crdb-cli crdb update`]():

   [code example]

Replace `<username>`, `<password>`, `<host>`, `<port>`, `<database-id>`, and `<crdb-guid>` with your actual values.

code:
curl -v -k -u <username>:<password> -X PUT \
  -H "Content-Type: application/json" \
  -d '{"sync": "enabled"}' \
  https://<host>:<port>/v1/bdbs/<database-id>

section_id: unrecoverable-syncer-errors

code: crdb-cli crdb update --crdb-guid <crdb-guid> --force


title: Configure distributed synchronization

title: Overview

text:
Replicated databases, such as [Replica Of]() and [Active-Active]() databases,
use proxy endpoints to synchronize database changes with the databases on other participating clusters.

To improve the throughput and lower the latency for synchronization traffic,
you can configure a replicated database to use distributed synchronization where any available proxy endpoint can manage synchronization traffic.

Every database by default has one proxy endpoint that manages client and synchronization communication with the database shards,
and that proxy endpoint is used for database synchronization.
This is called centralized synchronization.

To prepare a database to use distributed synchronization you must first make sure that the database [proxy policy]()
is defined so that either each node has a proxy endpoint or each primary (master) shard has a proxy endpoint.
After you have multiple proxies for the database,
you can configure the database synchronization to use distributed synchronization.

title: Configure distributed synchronization

role: configuration

text:
You may use the database name in place of `db:<ID>` in the following `rladmin` commands.

To configure distributed synchronization:

1. To check the proxy policy for the database, run: `rladmin status`

    The output of the status command shows the list of endpoints on the cluster and the proxy policy for the endpoint.

    [code example]

    If the proxy policy (also known as a _role_) is `single`, configure the policy to `all-nodes` or `all-master-shards` according to your needs with the command:

    [code example]

1. To configure the database to use distributed synchronization, run:

    [code example]

    To change back to centralized synchronization, run:

    [code example]

title: Verify database synchronization

text:
Use `rladmin` to verify a database synchronization role:

[code example]

The current database role is reported as the `syncer_mode` value:

[code example]

code:
ENDPOINTS:
    DB:ID       NAME      ID                        NODE          ROLE                                SSL
    db:1        db        endpoint:1:1              node:1        all-master-shards                   No

section_id: configure-distributed-synchronization

code: rladmin bind db db:<ID> endpoint <endpoint id> policy <all-master-shards|all-nodes>

code: rladmin tune db db:<ID> syncer_mode distributed

code: rladmin tune db db:<ID> syncer_mode centralized

code: rladmin info db db:<ID>

section_id: verify-database-synchronization

code:
$ rladmin info db db:<ID>
db:<ID> [<db_name>]:
  // (Other settings removed)
  syncer_mode: centralized


title: Configure database persistence

summary: How to configure database persistence with either an append-only file (AOF) or snapshots.

title: Overview

text:
Data is stored in RAM or a combination of RAM and flash memory ([Redis Flex and Auto Tiering]()), which risks data loss during process or server failures. Redis Software supports multiple methods to persist data to disk on a per-database basis to ensure data durability.

You can configure [persistence](https://redis.com/redis-enterprise/technology/durable-redis/) during database creation or by editing an existing database. Although the persistence model can be changed dynamically, the switch can take time depending on the database size and the models being switched.

title: Configure database persistence

text:
You can configure persistence when you [create a database](), or you can edit an existing database's configuration:

1. From the **Databases** list, select the database, then select **Configuration**.

1. Select **Edit**.

1. Expand the **Durability** section.

1. For **Persistence**, select an [option](#data-persistence-options) from the list.

1. Select **Save**.

title: Data persistence options

text:
There are six options for persistence in Redis Software:

|  **Options** | **Description** |
|  ------ | ------ |
|  None | Data is not persisted to disk at all. |
|  Append-only file (AOF) - fsync every write | Data is fsynced to disk with every write. |
|  Append-only file (AOF) - fsync every 1 sec | Data is fsynced to disk every second. |
|  Snapshot, every 1 hour | A snapshot of the database is created every hour. |
|  Snapshot, every 6 hours | A snapshot of the database is created every 6 hours. |
|  Snapshot, every 12 hours | A snapshot of the database is created every 12 hours. |

title: Select a persistence strategy

text:
When selecting your persistence strategy, you should take into account your tolerance for data loss and performance needs. There will always be tradeoffs between the two.
The fsync() system call syncs data from file buffers to disk. You can configure how often Redis performs an fsync() to most effectively make tradeoffs between performance and durability for your use case.
Redis supports three fsync policies: every write, every second, and disabled.

Redis also allows snapshots through RDB files for persistence. Within Redis Software, you can configure both snapshots and fsync policies.

For any high availability needs, use replication to further reduce the risk of data loss.

**For use cases where data loss has a high cost:**

Append-only file (AOF) - fsync every write - Redis Software sets the Redis directive `appendfsyncalways`.  With this policy, Redis will wait for the write and the fsync to complete prior to sending an acknowledgement to the client that the data has written. This introduces the performance overhead of the fsync in addition to the execution of the command. The fsync policy always favors durability over performance and should be used when there is a high cost for data loss.

**For use cases where data loss is tolerable only limitedly:**

Append-only file (AOF) - fsync every 1 sec - Redis will fsync any newly written data every second. This policy balances performance and durability and should be used when minimal data loss is acceptable in the event of a failure. This is the default Redis policy. This policy could result in between 1 and 2 seconds worth of data loss but on average this will be closer to one second.

If you use AOF for persistence, enable replication to improve performance. When both features are enabled for a database, the replica handles persistence, which prevents any performance impact on the master.

**For use cases where data loss is tolerable or recoverable for extended periods of time:**

- Snapshot, every 1 hour - Performs a full backup every hour.
- Snapshot, every 6 hour - Performs a full backup every 6 hours.
- Snapshot, every 12 hour - Performs a full backup every 12 hours.
- None - Does not back up or persist data at all.

title: Append-only file (AOF) vs snapshot (RDB)

text:
Now that you know the available options, to assist in making a decision
on which option is right for your use case, here is a table about the
two:

|  **Append-only File (AOF)** | **Snapshot (RDB)** |
|------------|-----------------|
|  More resource intensive | Less resource intensive |
|  Provides better durability (recover the latest point in time) | Less durable |
|  Slower time to recover (Larger files) | Faster recovery time |
|  More disk space required (files tend to grow large and require compaction) | Requires less resources (I/O once every several hours and no compaction required) |

title: Active-Active data persistence

text:
Active-Active databases support AOF persistence only.  Snapshot persistence is not supported for Active-Active databases.

If an Active-Active database is using snapshot persistence, use `crdb-cli` to switch to AOF persistence:

[code example]

title: Auto Tiering data persistence

text:
Auto Tiering flash storage is not considered persistent storage.

Flash-based databases are expected to hold larger datasets, and shard repair times can take longer after node failures. To better protect the database against node failures with longer repair times, consider enabling master and replica dual data persistence.

However, dual data persistence with replication adds some processor
and network overhead, especially for cloud configurations
with network-attached persistent storage, such as EBS-backed
volumes in AWS.

There may be times when performance is critical for your use case and
you don't want to risk data persistence adding latency.

You can enable or turn off data persistence on the master shards using the
following `rladmin` command:

[code example]

code:
crdb-cli crdb update --crdb-guid <CRDB_GUID> --default-db-config \
   '{"data_persistence": "aof", "aof_policy":"appendfsync-every-sec"}'

section_id: active-active-data-persistence

code: rladmin tune db <database_ID_or_name> master_persistence <disabled | enabled>

section_id: auto-tiering-data-persistence


title: Configure database defaults

summary: Cluster-wide policies that determine default settings when creating new databases.

title: Overview

text: Database defaults are cluster-wide policies that determine default settings when creating new databases.

title: Edit database defaults

text:
To edit default database configuration using the Cluster Manager UI:

1. On the **Databases** screen, select  to open a list of additional actions.

1. Select **Database defaults**.

1. Configure [database defaults](#db-defaults).

1. Select **Save**.

title: Database defaults {#db-defaults}

title: Endpoint configuration

text:
You can choose a predefined endpoint configuration to use the recommended database proxy and shards placement policies for your use case. If you want to set these policies manually instead, select **Custom** endpoint configuration.

| Endpoint configuration | Database proxy | Shards placement | Description |
|-----------|------------|----------------|------------------|------------|
| Enterprise clustering | Single | Dense | Sets up a single endpoint that uses DNS to automatically reflect IP address updates after failover or topology changes. |
| Using a load balancer | All nodes | Sparse | Configure Redis with a load balancer like HAProxy or Nginx for environments without DNS. |
| Multiple endpoints | All primary shards | Sparse | To set up multiple endpoints, enable **OSS Cluster API** in the database settings and ensure client support. Clients initially connect to the primary node to retrieve the cluster topology, which allows direct connections to individual Redis proxies on each node. |
| Custom | Single, all primary shards, or all nodes | Dense or sparse | Manually choose default database proxy and shards placement policies. |

title: Database proxy

text:
Redis Software uses [proxies]() to manage and optimize access to database shards. Each node in the cluster runs a single proxy process, which can be active (receives incoming traffic) or passive (waits for failovers).

You can configure default [proxy policies]() to determine which nodes' proxies are active and bound to new databases by default.

To configure the default database proxy policy using the Cluster Manager UI:

1. [**Edit database defaults**](#edit-database-defaults).

1. Select a predefined [**Endpoint Configuration**](#endpoint-configuration) to use a recommended database proxy policy, or choose **Custom** to set the policy manually. Changing the database proxy default in the Cluster Manager UI affects both sharded and non-sharded proxy policies.

#### Non-sharded proxy policy

To configure the default proxy policy for non-sharded databases, use one of the following methods:

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

#### Sharded proxy policy

To configure the default proxy policy for sharded databases, use one of the following methods:

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

title: Shards placement

text:
The default [shard placement policy]() determines the distribution of database shards across nodes in the cluster.

Shard placement policies include:

- `dense`: places shards on the smallest number of nodes.

- `sparse`: spreads shards across many nodes.

To configure default shard placement, use one of the following methods:

- Cluster Manager UI:

    1. [**Edit database defaults**](#edit-database-defaults).

    1. Select a predefined [**Endpoint Configuration**](#endpoint-configuration) to use a recommended shards placement policy, or choose **Custom** to set the policy manually.

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

title: Database version

text:
New databases use the default Redis database version unless you select a different **Database version** when you [create a database]() in the Cluster Manager UI or specify the `redis_version` in a [create database REST API request]().

To configure the Redis database version, use one of the following methods:

- Cluster Manager UI: Edit **Database version** in [**Database defaults**](#edit-database-defaults)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

title: Internode encryption

text:
Enable [internode encryption]() to encrypt data in transit between nodes for new databases by default.

To enable or turn off internode encryption by default, use one of the following methods:

- Cluster Manager UI: Edit **Internode Encryption** in [**Database defaults**](#edit-database-defaults)

- [rladmin tune cluster]():

    [code example]

- [Update cluster policy]() REST API request:

    [code example]

code: rladmin tune cluster default_non_sharded_proxy_policy { single | all-master-shards | all-nodes }

section_id: database-proxy

code:
PUT /v1/cluster/policy
    { "default_non_sharded_proxy_policy": "single | all-master-shards | all-nodes" }

code: rladmin tune cluster default_sharded_proxy_policy { single | all-master-shards | all-nodes }

code:
PUT /v1/cluster/policy
    { "default_sharded_proxy_policy": "single | all-master-shards | all-nodes" }

code: rladmin tune cluster default_shards_placement { dense | sparse }

section_id: shards-placement

code:
PUT /v1/cluster/policy
    { "default_shards_placement": "dense | sparse" }

code: rladmin tune cluster default_redis_version <x.y>

section_id: database-version

code:
PUT /v1/cluster/policy
    { "default_provisioned_redis_version": "x.y" }

code: rladmin tune cluster data_internode_encryption { enabled | disabled }

section_id: internode-encryption

code:
PUT /v1/cluster/policy
    { "data_internode_encryption": <boolean> }


title: Manage database tags

summary: Manage tags for databases in a Redis Software cluster.

title: Overview

text:
You can create custom tags to categorize databases in a Redis Software cluster.

The **Databases** screen shows tags for each database in the list.

title: Add database tags

text:
You can add tags when you [create a database]() or [edit an existing database's configuration]().

To add tags to a database using the Cluster Manager UI:

1. While in edit mode on the database's configuration screen, click **Add tags**.

1. Enter a key and value for the tag. Keys and values previously used by existing tags will appear as suggestions.

1. To add additional tags, click **Add tag**.

1. After you finish adding tags, click **Done** to close the tag manager.

1. Click **Create** or **Save**.

title: Edit database tags

text:
To edit a database's existing tags using the Cluster Manager UI:

1. Go to the database's **Configuration** screen, then click **Edit**.

1. Next to the existing **Tags**, click .

1. Edit or delete existing tags, or click **Add tag** to add new tags.

1. After you finish editing tags, click **Done** to close the tag manager.

1. Click **Save**.