rsyslog logging

Log concepts

Redis Software logs information from a variety of components in response to actions and events that occur within the cluster.

In some cases, a single action, such as removing a node from the cluster, may actually consist of several events. These actions may generate multiple log entries.

All log entries displayed in the Cluster Manager UI are also written to `syslog`.  You can configure `rsyslog` to monitor `syslog`. Enabled alerts are logged to `syslog` and appear with other log entries.

You can also [manage your logs]() with a remote logging server and log rotation.

Types of log entries

Log entries are categorized into events and alerts. Both types of entries appear in the logs, but alert log entries also include a boolean `"state"` parameter that indicates whether the alert is enabled or disabled.

Log entries include information about the specific event that occurred. See the log entry tables for [alerts and events]() for more details.

Severity

You can also configure `rsyslog` to add other information, such as the event severity.

Since `rsyslog` entries do not include severity by default, you can follow these steps to enable it:

1. Add the following line to `/etc/rsyslog.conf`:
    [code example]

2. Modify `$ActionFileDefaultTemplate` to use your new template `$ActionFileDefaultTemplateTraditionalFormatWithPRI`

3. Save these changes and restart `rsyslog` to apply them

You can see the log entries for alerts and events in the `/var/log/messages` file.

**Command components:**

- `%pritext%` adds the severity
- `%timegenerated%` adds the timestamp
- `%HOSTNAME%` adds the machine name
- `%syslogtag%` adds the Redis Software message. See the [log entry structure](#log-entry-structure) section for more details.
- `%msg:::droplastlf%n` removes duplicated log entries

Log entry structure

The log entries have the following basic structure:

    event_log[<process id>]:{<list of key-value pairs in any order>}

- **event_log**: Plain static text is always shown at the beginning of the entry.
- **process id**: The ID of the logging process
- **list of key-value pairs in any order**: A list of key-value pairs that describe the specific event. They can appear in any order. Some key-value pairs are always shown, and some appear depending on the specific event.
    - **Key-value pairs that always appear:**
        - `"type"`: A unique code name for the logged event. For the list of codenames, see the [logged alerts and events]() tables.
        - `"object"`: Defines the object type and ID (if relevant) of the object this event relates to, such as cluster, node with ID, BDB with ID, etc. Has the format of <nobr>`<object type>[:<id>]`</nobr>.
        - `"time"`: Unix epoch time but can be ignored in this context.
    - **Key-value pairs that might appear depending on the specific entry:**
        - `"state"`: A boolean where `true` means the alert is enabled, and `false` means the alert is disabled. This is only relevant for alert log entries.
        - `"global_threshold"`: The value of a threshold for alerts related to cluster or node objects.
        - `"threshold"`: The value of a threshold for alerts related to a BDB object

Log entry samples

This section provides examples of log entries that include the [`rsyslog` configuration](#severity) to add the severity, timestamp, and machine name.

Ephemeral storage passed threshold

#### "Alert on" log entry sample

[code example]

In this example, the storage utilization on node 1 reached the value of ~90%, which triggered the alert for "Ephemeral storage has reached 70% of its capacity."

**Log entry components:**

- `daemon.warning` - Severity of entry is `warning`
- `Jun 14 14:49:20` - The timestamp of the event
- `node1`: Machine name
- `event_log` - Static text that always appears
- `[3464]` - Process ID
- `"storage_util":90.061643120001` - Current ephemeral storage utilization
- `"global_threshold":"70"` - The user-configured threshold above which the alert is raised
- `"object":"node:1"` - The object related to this alert
- `"state":true` - Current state of the alert
- `"time":1434282560` - Can be ignored
- `"type":"ephemeral_storage"` - The code name of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

This log entry is an example of when the alert for the node with ID 1 "Ephemeral storage has reached 70% of its capacity" has been turned off as result of storage utilization reaching the value of ~60%.

**Log entry components**:

- `daemon.info` - Severity of entry is `info`
- `Jun 14 14:51:35` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[3464]` - Process ID
- `"storage_util":60.051723520008` - Current ephemeral storage utilization
- `"global_threshold":"70"` - The user configured threshold above which the alert is raised (70% in this case)
- `"object":"node:1"` - The object related to this alert
- `"state":false` - Current state of the alert
- `"time":1434283480` - Can be ignored
- `"type":"ephemeral_storage"` - The code name identifier of this specific event. See [logged alerts and events]() for more details.

Odd number of nodes with a minimum of three nodes alert

#### "Alert on" log entry sample

[code example]

This log entry is an example of when the alert for "True high availability requires an odd number of nodes with a minimum of three nodes" has been turned on as result of the cluster having only one node.

**Log entry components:**

- `daemon.warning` - Severity of entry is warning
- `Jun 14 15:25:00` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[8310]` - Process ID
- `"object":"cluster"` - The object related to this alert
- `"state":true` - Current state of the alert
- `"time":1434284700` - Can be ignored
- `"node_count":1` - The number of nodes in the cluster
- `"type":"even_node_count"` - The code name identifier of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

This log entry is an example of when the alert for "True high availability requires an odd number of nodes with a minimum of three nodes" has been turned off as result of the cluster having 3 nodes.

**Log entry components:**

- `daemon.warning` - Severity of entry is warning
- `Jun 14 15:30:40` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[8310]` - Process ID
- `"object":"cluster"` - The object related to this alert
- `"state":false` - Current state of the alert
- `"time":1434285200` - Can be ignored
- `"node_count":3` - The number of nodes in the cluster
- `"type":"even_node_count"` - The code name of this specific event. See [logged alerts and events]() for more details.

#### "Alert on" log entry sample

[code example]

This log entry is an example of when the alert for "Node has insufficient disk space for AOF rewrite" has been turned on as result of not having enough persistent storage disk space for AOF rewrite purposes. It is missing 604602126 bytes.

**Log entry components:**

- `daemon.err` - Severity of entry is error
- `Jun 15 13:51:23` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[34252]` - Process ID
- `"used":23457188` - The amount of disk space in bytes currently used for AOF files
- `"missing":604602126` - The amount of disk space in bytes that is currently missing for AOF rewrite purposes
- `"object":"node:1″` - The object related to this alert
- `"free":9867264` - The amount of disk space in bytes that is currently
    free
- `"needed":637926578` - The amount of total disk space in bytes that is needed for AOF rewrite purposes
- `"state":true` - Current state of the alert
- `"time":1434365483` - Can be ignored
- `"disk":705667072` - The total size in bytes of the persistent storage
- `"type":"insufficient_disk_aofrw"` - The code name of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

**Log entry components:**

- `daemon.info` - Severity of entry is info
- `Jun 15 13:51:11` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[34252]` - Process ID
- `"used":0` - The amount of disk space in bytes currently used for AOF files
- `"missing":‐21614592` - The amount of disk space in bytes that is currently missing for AOF rewrite purposes. In this case, it is not missing because the number is negative.
- `"object":"node:1″` - The object related to this alert
- `"free":21614592` - The amount of disk space in bytes that is currently free
- `"needed":0` - The amount of total disk space in bytes that is needed for AOF rewrite purposes. In this case, no space is needed.
- `"state":false` - Current state of the alert
- `"time":1434365471` - Can be ignored
- `"disk":705667072` - The total size in bytes of the persistent storage
- `"type":"insufficient_disk_aofrw"` - The code name of this specific event. See [logged alerts and events]() for more details.

$template TraditionalFormatWithPRI,"%pri-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

daemon.warning: Jun 14 14:49:20 node1 event_log[3464]:
{
    "storage_util": 90.061643120001,
    "global_threshold": "70",
    "object": "node:1",
    "state": true,
    "time": 1434282560,
    "type": "ephemeral_storage"
}

daemon.info: Jun 14 14:51:35 node1 event_log[3464]:
{
    "storage_util":60.051723520008,
    "global_threshold": "70",
    "object": "node:1",
    "state":false,
    "time": 1434283480,
    "type": "ephemeral_storage"
}

daemon.warning: Jun 14 15:25:00 node1 event_log[8310]:
{
    "object":"cluster",
    "state": true,
    "time": 1434284700,
    "node_count": 1,
    "type":"even_node_count"
}

daemon.warning: Jun 14 15:30:40 node1 event_log[8310]:
{
    "object":"cluster",
    "state": false,
    "time": 1434285200,
    "node_count": 3,
    "type":"even_node_count"
}


Maintenance mode for cluster nodes

Summary: Prepare a cluster node for maintenance.

Overview

Use maintenance mode to prevent data loss during hardware patching or operating system maintenance on Redis Software servers. When maintenance mode is on, all shards move off of the node under maintenance and migrate to another available node.

Activate maintenance mode

When you activate maintenance mode, Redis Software does the following:

1. Checks whether the cluster has enough active nodes to maintain quorum if one goes down. If taking a node offline would cause quorum loss, maintenance mode will not activate.

This check does not consider if other nodes are already in maintenance mode. Do not put the majority of nodes into maintenance mode and restart them simultaneously, as this can cause the cluster to lose quorum and lead to data loss.

1. If no maintenance mode snapshots already exist or if you use `overwrite_snapshot` when you activate maintenance mode, Redis Software creates a new node snapshot that records the node's shard and endpoint configuration.

1. Marks the node as a quorum node to prevent shards and endpoints from migrating to it.

    At this point, [`rladmin status`]() displays the node's shards field in yellow, which indicates that shards cannot migrate to the node.

1. Migrates shards and binds endpoints to other nodes, when space is available.

Maintenance mode does not demote a master node by default. The cluster elects a new master node when the original master node restarts.

Add the `demote_node` option to the `rladmin` command to [demote a master node](#demote-a-master-node) when you activate maintenance mode.

To activate maintenance mode for a node, run the following command:

[code example]

You can start server maintenance if:

- All shards and endpoints have moved to other nodes

- Enough nodes are still online to maintain quorum

Prevent replica shard migration

If you do not have enough resources available to move all of the shards to other nodes, you can turn maintenance mode on without migrating the replica shards.

Before you prevent replica shard migration during maintenance mode, consider the following effects:

- Replica shards remain on the node during maintenance.

- If the maintenance node fails, the master shards do not have replica shards to maintain data redundancy and high availability.

- Replica shards that remain on the node can still be promoted during failover to preserve availability.

To activate maintenance mode without replica shard migration, run:

[code example]

Demote a master node

If maintenance might affect connectivity to the master node, you can demote the master node when you  activate maintenance mode.  This lets the cluster elect a new master node.

To demote a master node when activating maintenance mode, run:

[code example]

Verify maintenance mode activation

To verify maintenance mode for a node, use `rladmin status` and review the node's shards field.  If that value is displayed in yellow (shown earlier), then the node is in maintenance mode.

Avoid activating maintenance mode when it is already active.  Maintenance mode activations stack.  If you activate maintenance mode for a node that is already in maintenance mode, you will have to deactivate maintenance mode twice in order to restore full functionality.

Deactivate maintenance mode

When you deactivate maintenance mode, Redis Software:

1. Loads a [specified node snapshot](#specify-a-snapshot) or defaults to the latest maintenance mode snapshot.

1. Unmarks the node as a quorum node to allow shards and endpoints to migrate to the node.

1. Restores the shards and endpoints that were in the node at the time of the snapshot.

1. Deletes the snapshot.

To deactivate maintenance mode after server maintenance, run:

[code example]

By default, a snapshot is required to deactivate maintenance mode.  If the snapshot cannot be restored, deactivation is cancelled and the node remains in maintenance mode.  In such events, it may be necessary to [reset node status](#reset_node_status).

Specify a snapshot

When you turn off maintenance mode, you can restore the node configuration from a maintenance mode snapshot or any snapshots previously created by [`rladmin node <node_id> snapshot create`](). If you do not specify a snapshot, Redis Software uses the latest maintenance mode snapshot by default.

To get a list of available snapshots, run:

[code example]

To specify a snapshot when you turn maintenance mode off, run:

[code example]

If an error occurs when you turn on maintenance mode, the snapshot is not deleted.
When you rerun the command, use the snapshot from the initial attempt since it contains the original state of the node.

Skip shard restoration

You can prevent the migrated shards and endpoints from returning to the original node after you turn off maintenance mode.

To turn maintenance mode off and skip shard restoration, run:

[code example]

Reset node status

In extreme cases, you may need to reset a node's status.  Run the following commands to do so:

[code example]

Use these commands with caution.  For best results, contact Support before running these commands.

Cluster status example

This example shows how the output of [`rladmin status`]() changes when you turn on maintenance mode for a node.

The cluster status before turning on maintenance mode:

[code example]

The cluster status after turning on maintenance mode:

[code example]

After turning on maintenance mode for node 2, Redis Software saves a snapshot of its configuration and then moves its shards and endpoints to node 3.

Now node 2 has `0/0` shards because shards cannot migrate to it while it is in maintenance mode.

Maintenance mode REST API

You can also turn maintenance mode on or off using [REST API requests]() to [<nobr>POST `/nodes/{node_uid}/actions/{action}`</nobr>]().

Activate maintenance mode (REST API)

Use <nobr>`POST /nodes/{node_uid}/actions/maintenance_on`</nobr> to activate maintenance mode:

[code example]

You can set `evict_ha_replica` and `evict_active_active_replica` to `false` to [prevent replica shard migration](#prevent-replica-shard-migration).

The `maintenance_on` request returns a JSON response body:

[code example]

Deactivate maintenance mode (REST API)

Use <nobr>`POST /nodes/{node_uid}/actions/maintenance_off`</nobr> deactivate maintenance mode:

[code example]

The `skip_shards_restore` boolean flag allows the `maintenance_off` action to [skip shard restoration](#skip-shard-restoration) when set to `true`.

The `maintenance_off` request returns a JSON response body:

[code example]

Track action status

You can send a request to [<nobr>GET `/nodes/{node_uid}/actions/{action}`</nobr>]() to track the [status]() of the `maintenance_on` and `maintenance_off` actions.

This request returns the status of the `maintenance_on` action:

[code example]

The response body:

[code example]

rladmin node <node_id> maintenance_mode on overwrite_snapshot

rladmin node <node_id> maintenance_mode on evict_ha_replica disabled evict_active_active_replica disabled

rladmin node <node_id> maintenance_mode on demote_node

rladmin node <node_id> maintenance_mode off snapshot_name <snapshot_name>

rladmin node <node_id> maintenance_mode off skip_shards_restore

$ rladmin tune node <node_id> max_listeners 100
$ rladmin tune node <node_id> quorum_only disabled

redislabs@rp1_node1:/opt$ rladmin status
CLUSTER NODES:
NODE:ID   ROLE     ADDRESS       EXTERNAL_ADDRESS     HOSTNAME    SHARDS
*node:1   master   172.17.0.2                         rp1_node1   2/100
node:2    slave    172.17.0.4                         rp3_node1   2/100
node:3    slave    172.17.0.3                         rp2_node1   0/100

redislabs@rp1_node1:/opt$ rladmin node 2 maintenance_mode on
Performing maintenance_on action on node:2: 0%
created snapshot NodeSnapshot<name=maintenance_mode_2019-03-14_09-50-59,time=None,node_uid=2>

node:2 will not accept any more shards
Performing maintenance_on action on node:2: 100%
OK
redislabs@rp1_node1:/opt$ rladmin status
CLUSTER NODES:
NODE:ID   ROLE     ADDRESS       EXTERNAL_ADDRESS     HOSTNAME    SHARDS
*node:1   master   172.17.0.2                         rp1_node1   2/100
node:2    slave    172.17.0.4                         rp3_node1   0/0
node:3    slave    172.17.0.3                         rp2_node1   2/100

POST https://<hostname>:9443/v1/nodes/<node_id>/actions/maintenance_on
{
    "overwrite_snapshot": true,
    "evict_ha_replica": true,
    "evict_active_active_replica": true
}

POST https://<hostname>:9443/v1/nodes/<node_id>/actions/maintenance_off
{ "skip_shards_restore": false }


Set up a new cluster

Summary: How to set up a new cluster using the management UI.

Content

A Redis Software cluster typically consists of several nodes.
For production deployments, we recommend an uneven number of nodes, with a minimum of three.

In a cluster that consists of only one node, some features and capabilities are not enabled,
such as database replication that provides high availability.

To set up a new cluster, you must first [install the Redis Software package]()
and then set up the cluster as described below.
After the cluster is created you can [add multiple nodes to the cluster]().

To create a cluster:

1. In a browser, go to `https://<name or IP address of the machine with Redis Software installed>:8443`.
    For example, if you installed Redis Software on a machine with IP address 10.0.1.34, go to <https://10.0.1.34:8443>.

- The management UI uses a [self-signed certificate for TLS encryption]().
- If the machine has both an internal IP address and an external IP address, use the external IP address to access the setup UI.

1. Select **Create new cluster**.

1. Enter an email and password for the administrator account, then select **Next** to proceed to cluster setup.

1. Enter your cluster license key if you have one. Otherwise, the cluster uses the trial license by default.

1. In the **Configuration** section:

    1. For **FQDN (Fully Qualified Domain Name)**, enter a unique name for the cluster.

        See the [instructions for DNS setup]()
        to make sure your cluster is reachable by name.

    1. Choose whether to [**Enable private & public endpoints support**]().

    1. Choose whether to [**Enable rack-zone awareness**]().

1. Click **Next**.

1. Configure storage and network settings:

    1. Enter a path for [*Ephemeral storage*](), or leave the default path.

    1. Enter a path for [*Persistent storage*](),
        or leave the default path.

    1. To enable [*Redis Flex or Auto Tiering*](),
        select **Enable flash storage** and enter the path to the flash storage.

    1. If the cluster is configured to support [rack-zone awareness](), set the **Rack-zone ID** for the new node.

    1. If your machine has multiple IP addresses, assign a single IPv4 type address for **Node-to-node communication (internal traffic)** and multiple IPv4/IPv6 type addresses for **External traffic**.

1. Select **Create cluster**.

1. Click **OK** to confirm that you are aware of the replacement of the HTTPS TLS certificate on the node,
    and proceed through the browser warning.

After a short wait, your cluster is created and you can sign in to the Cluster Manager UI.

You can now access any of the management capabilities, including:

- [Creating a new database]()
- [Joining a new node to a cluster]()


Disk sizing for heavy write scenarios

Summary: Sizing considerations for persistent disk space for heavy throughput databases.

Overview

In extreme write scenarios when append-only files (AOF) are enabled, the AOF rewrite process can require considerably more disk space for [database persistence]().

For disk size requirements for standard usage, see [Hardware requirements]().

Estimate required disk space

To estimate the required persistent disk space for AOF rewrite purposes in extreme write scenarios, use the following formula:

**X (1 + 3Y + Y²)**

Where:
- **X** = size of each shard in GB
- **Y** = number of shards

Examples

The following examples show how to calculate the persistent disk space required for heavy write scenarios for different database configurations, where:

- **Database size** is the memory limit configured for the database.

- **Number of shards** is the total number of shards (primary shards + replica shards).

Example 1

- Database size = 10 GB
- Number of shards = 4

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 73 GB of required disk space.

Example 2

- Database size = 10 GB
- Number of shards = 16

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 191 GB of required disk space.

Example 3

- Database size = 40 GB
- Number of shards = 5

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Required disk space: 328 GB.

Example 4

- Database size = 40 GB
- Number of shards = 15

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 724 GB of required disk space.

Shard size = database size / number of shards
               = 10 GB / 4 shards
               = 2.5 GB per shard

Shard size = database size / number of shards
               = 10 GB / 16 shards
               = 0.625 GB per shard

Shard size = database size / number of shards
               = 40 GB / 5 shards
               = 8 GB per shard

Shard size = database size / number of shards
               = 40 GB / 15 shards
               = 2.67 GB per shard


Optimize clusters

Summary: Configuration changes and information you can use to optimize your performance and memory usage.

Summary: Use the `memtier_benchmark` tool to perform a performance benchmark of Redis Software.

Benchmarking Redis Software

Disk sizing for heavy write scenarios

Summary: Optimize your cluster environments for better performance.

Cluster environment optimization

Summary: Turn off services to free memory and improve performance.

Turn off services to free system memory

Summary: Use the Redis OSS Cluster API to improve performance and keep applications current with cluster topology changes.

OSS Cluster API

Summary: Use the wait command to take full advantage of Redis Software's replication-based durability.

Use the WAIT command to improve data safety and durability


Benchmarking Redis Software

Overview

Use the `memtier_benchmark` tool to perform a performance benchmark of Redis Software.

Prerequisites:

- Redis Software installed
- A cluster configured
- A database created

For help with the prerequisites, see the [Redis Software quickstart]().

It is recommended to run memtier_benchmark on a separate node that is
not part of the cluster being tested. If you run it on a node of the
cluster, be mindful that it affects the performance of both the
cluster and memtier_benchmark.

[code example]

This command instructs memtier_benchmark to connect to your Redis
Enterprise database and generates a load doing the following:

- A 50/50 Set to Get ratio
- Each object has random data in the value

Populate a database with testing data

If you need to populate a database with some test data for a proof of
concept, or failover testing, etc. here is an example for you.

[code example]

This command instructs memtier_benchmark to connect to your Redis
Enterprise database and generates a load doing the following:

- Write objects only, no reads
- A 500 byte object
- Each object has random data in the value
- Each key has a random pattern, then a colon, followed by a
    random pattern.

Run this command until it fills up your database to where you want it
for testing. The easiest way to check is on the database metrics page.

Another use for memtier_benchmark is to populate a database with data
for failure testing.

/opt/redislabs/bin/memtier_benchmark -s $DB_HOST -p $DB_PORT -a $DB_PASSWORD -t 4 -R --ratio=1:1

/opt/redislabs/bin/memtier_benchmark -s $DB_HOST -p $DB_PORT -a $DB_PASSWORD -R -n allkeys -d 500 --key-pattern=P:P --ratio=1:0


Cluster environment optimization

Overview

Redis Software uses various algorithms to optimize
performance. As part of this process, Redis Software examines usage
and load to adjust its runtime configuration. Depending
on your specific usage and load, Redis Software might take some
time to adjust for optimal performance.

To ensure optimal performance, you must run your workload several times
and for a long duration until performance stabilizes.

Failure detection sensitivity policies

You can optimize your cluster's thresholds and timeouts for different environments using the `failure_detection_sensitivity` cluster policy:

- `high` (previously known as `local-network watchdog_profile`) – high failure detection sensitivity, lower thresholds, and faster failure detection and failover

- `low` (previously known as `cloud watchdog_profile`) – low failure detection sensitivity and higher tolerance for latency variance (also called network jitter)

Depending on which policy you choose, Redis Software uses different
thresholds to make operation-related decisions.

The default policy is `high` failure detection sensitivity for `local-network` environments. If you are
running Redis Software in a cloud environment, you should change the
configuration.

Change failure detection sensitivity

To change the cluster's `failure_detection_sensitivity`, run one of the following [`rladmin`]() commands.

- For Redis Software version 6.4.2-69 and later, run:

    [code example]

- For Redis Software version 6.4.2-61 and earlier, run:

    [code example]

If Redis Software still
does not meet your performance expectations after following these instructions, [contact support](https://redis.com/company/support/) for help optimizing your environment.

rladmin tune cluster failure_detection_sensitivity [ low | high ]

rladmin tune cluster watchdog_profile [ cloud | local-network ]


OSS Cluster API

Content

You can use the Redis OSS Cluster API along with other Redis Enterprise Software high availability
to get high performance with low latency
and let applications stay current with cluster topology changes, including add node, remove node, and node failover.

For more about working with the OSS Cluster API in Redis Software, see [Enable OSS Cluster API]().

To learn how to enable OSS Cluster API in Redis Cloud, see [Clustering Redis databases]().

To enable OSS Cluster API in Kubernetes, see [Enable cluster-aware clients (OSS Cluster API)]().


Turn off services to free system memory

Content

The Redis Software cluster nodes host a range of services that support the cluster processes.
In most deployments, either all of these services are required,
or there are enough memory resources on the nodes for the database requirements.

In a deployment with limited memory resources, certain services can be disabled from API endpoint to free system memory or using the `rladmin` command.
Before you turn off a service, make sure that your deployment does not depend on that service.
After you turn off a service, you can re-enable in the same way.

The services that you can turn off are:

- RS Admin Console - `cm_server`
- Logs in CSV format - `stats_archiver`
- [LDAP authentication]() - `saslauthd`
- [Discovery service]()- `mdns_server`, `pdns_server`
- [Active-Active databases]() - `crdb_coordinator`, `crdb_worker`
- Alert Manager - `alert_mgr` (For best results, disable only if you have an alternate alert system.)

To turn off a service with the `rladmin cluster config` command, use the `services` parameter and the name of the service, followed by `disabled`.
[code example]

To turn off a service with the API, use the [`PUT /v1/services_configuration`]() endpoint
with the name of the service and the operating mode (enabled/disabled) in JSON format.

For example:
- To turn off the Redis Software Cluster Manager UI, use this PUT request:

    [code example]

- To turn off the CRDB services and enable the `stats_archiver` for cluster component statistics, use this PUT request:

    [code example]

rladmin cluster config
        [ services <service_name> <enabled | disabled> ]

PUT https://[host][:9443]/v1/cluster/services_configuration
    '{
        "cm_server":{
            "operating_mode":"disabled"
        }
    }'

PUT https://[host][:9443]/v1/cluster/services_configuration
    '{
        "crdb_coordinator":{
            "operating_mode":"disabled"
        },
        "crdb_worker":{
            "operating_mode":"disabled"
        },
        "stats_archiver":{
            "operating_mode":"enabled"
        }
    }'


Overview

Redis Software comes with the ability to replicate data
to another replica for high availability and persist in-memory data on
disk permanently for durability. With the [`WAIT`]() command, you can
control the consistency and durability guarantees for the replicated and
persisted database.

Non-blocking Redis write operation

Any updates that are issued to the database are typically performed with the following flow:

1. Application issues a write.
1. Proxy communicates with the correct primary shard in the system that contains the given key.
1. The acknowledgment is sent to proxy after the write operation completes.
1. The proxy sends the acknowledgment back to the application.

Independently, the write is communicated from the primary shard to the replica, and
replication acknowledges the write back to the primary shard. These are steps 5 and 6.

Independently, the write to a replica is also persisted to disk and
acknowledged within the replica. These are steps 7 and 8.

Blocking write operation on replication

With the [`WAIT`]() or [`WAITAOF`]() commands, applications can ask to wait for
acknowledgments only after replication or persistence is confirmed on
the replica. The flow of a write operation with `WAIT` or `WAITAOF` is:

1. The application issues a write.
1. The proxy communicates with the correct primary shard in the system that contains the given key.
1. Replication communicates the update to the replica shard.
1. If using `WAITAOF` and the AOF every write setting, the replica persists the update to disk before sending the acknowledgment.
1. The acknowledgment is sent back from the replica all the way to the proxy with steps 5 to 8.

The application only gets the acknowledgment from the write after durability is achieved with replication to the replica for `WAIT` or `WAITAOF` and to the persistent storage for `WAITAOF` only.

The `WAIT` command always returns the number of replicas that acknowledged the write commands sent by the current client before the `WAIT` command, both in the case where the specified number of replicas are reached, or when the timeout is reached. In Redis Software, the number of replicas is always 1 for databases with high availability enabled.

See the [`WAITAOF`]() command for details for enhanced data safety and durability capabilities introduced with Redis 7.2.


Remove a node

Summary: Remove a node from your Redis Software cluster.

Overview

You might want to remove a node from a Redis Software cluster for one of the following reasons:

- To [permanently remove a node](#permanently-remove-a-node) if you no longer need the extra capacity.
- To [replace a faulty node](#replace-a-faulty-node) with a healthy node.
- To [replace a healthy node](#replace-a-healthy-node) with a different node.

You can configure [email alerts from the cluster]() to notify you of cluster changes, including when a node is removed.

Read through these explanations thoroughly before taking
any action.

Permanently remove a node

Permanently removing a node means you are decreasing cluster capacity.
Before trying to remove a node, make sure that the cluster has enough
capacity for all resources without that node, otherwise you cannot remove the node.

If there is not enough capacity in the cluster to facilitate removing
the node, you can either delete databases or add another node instead of
the one you would like to remove.

During the removal process, the cluster migrates all resources from the
node being removed to other nodes in the cluster. In order to ensure
database connectivity, and database high availability (when replication
is enabled), the cluster first creates replacement shards or endpoints
on one of the other nodes in the cluster, initiates failover as needed,
and only then removes the node.

If a cluster has only two nodes (which is not recommended for production
deployments) and some databases have replication enabled, you cannot remove a node.

Replace a faulty node

If the cluster has a faulty node that you would like to replace, you
only need to add a new node to the cluster. The cluster recognizes the
existence of a faulty node and automatically replaces the faulty node
with the new node.

For guidelines, refer to [Replacing a faulty
node]().

Replace a healthy node

If you would like to replace a healthy node with a different node, you
must first add the new node to the cluster, migrate all the resources
from the node you would like to remove, and only then remove the node.

For further guidance, refer to [adding a new node to a
cluster]().

You can migrate resources by using the `rladmin` command-line interface
(CLI). For guidelines, refer to [`rladmin` command-line interface
(CLI)]().

The [DNS records]() must be updated each time a node is added or replaced.

Remove a node

To remove a node using the Cluster Manager UI:

1. On the **Nodes** screen, click  for the node you want to remove.

1. Select **Remove node** from the list.

1. If there are shards on the node, select **Confirm node removal**.

    If the node has no shards, the **Confirm node removal** checkbox does not appear.

1. Click **Remove**.

    Redis Software examines the node and the cluster, then takes the actions required to remove the node, such as migrating shards to other nodes. After the process finishes, the node is no longer shown in the UI.

At any point, you can click the **Cancel removal** button to stop the process. When cancelled, the current internal action is completed, and then the process stops.

To remove a node using the REST API, use [`POST /v1/nodes/<node_id>/actions/remove`]().

By default, the remove node action completes after all resources migrate off the removed node. Node removal does not wait for migrated shards' persistence files to be created on the new nodes.

To change node removal to wait for the creation of new persistence files for all migrated shards, set `wait_for_persistence` to `true` in the request body or [update the cluster policy]() `persistent_node_removal` to `true` to change the cluster's default behavior.

For example:

[code example]

If you need to add a removed node back to the cluster,
you must [uninstall]()
and [reinstall]() the software on that node.

POST https://<hostname>:9443/v1/nodes/<node_id>/actions/remove
{
    "wait_for_persistence": true
}


Replace a cluster node

Summary: Replace a node in your cluster that is down.

Content

A failed node will appear as `Down` () in the **Nodes** list.

To replace a failed node:

1. Prepare a new node identical to the old one.

1.  Install and
    configure Redis Software on the node. See [Install and setup]() for more information.

If you are using [Redis Flex or Auto Tiering](), make sure the required flash storage is set up on this new node.

1. [Add the node]() to the cluster. Make sure the new node has as much available memory as the faulty
    node.

    If the new node does not have enough memory, you will be prompted to add a node with enough memory.

1. A message will appear informing you that the cluster has a faulty node
    and that the new node will replace the faulty node.

- If there is a faulty node in the cluster to which you are adding a node, Redis Software will use the new node to replace the faulty one.
- Any existing [DNS records]() must be updated
each time a node is added or replaced.


Enable causal consistency

Summary: Enable causal consistency in an Active-Active database.

Overview

When you enable causal consistency in Active-Active databases,
the order of operations on a specific key are maintained across all Active-Active database instances.

For example, if operations A and B were applied on the same key and the effect of A was observed by the instance that initiated B before B was applied to the key.
All instances of an Active-Active database would then observe the effect of A before observing the effect of B.
This way, any causal relationship between operations on the same key is also observed and maintained by every replica.

Enable causal consistency

When you create an Active-Active database, you can enable causal consistency in the Cluster Manager UI:

1. In the **Participating clusters** section of the **Create Active-Active database** screen, locate **Causal Consistency**:

1. Click **Change** to open the **Causal Consistency** dialog.

1. Select **Enabled**:

1. Click **Change** to confirm your selection.

After database creation, you can only turn causal consistency on or off using the REST API or `crdb-cli`.
The updated setting only affects commands and operations received after the change.

Causal consistency side effects

When the causal consistency option is enabled, each instance maintains the order of operations it received from another instance
and relays that information to all other N-2 instances,
where N represents the number of instances used by the Active-Active database.

As a result, network traffic is increased by a factor of (N-2).
The memory consumed by each instance and overall performance are also impacted when causal consistency is activated.


Connect to your Active-Active databases

Summary: How to connect to an Active-Active database using redis-cli or a sample Python application.

Overview

With the Redis database created, you are ready to connect to your
database to store data. You can use one of the following ways to test
connectivity to your database:

- Connect with redis-cli, the built-in command-line tool
- Connect with a _Hello World_ application written in Python

Remember we have two member Active-Active databases that are available for connections and
concurrent reads and writes. The member Active-Active databases are using bi-directional
replication to for the global Active-Active database.

Connecting using redis-cli {#connecting-using-rediscli}

redis-cli is a simple command-line tool to interact with redis database.

1. To use redis-cli on port 12000 from the node 1 terminal, run:

    [code example]

1. Store and retrieve a key in the database to test the connection with these
    commands:

    - `set key1 123`
    - `get key1`

    The output of the command looks like this:

    [code example]

1. Enter the terminal of node 1 in cluster 2, run the redis-cli, and
   retrieve key1.

    The output of the commands looks like this:

    [code example]

Connecting using _Hello World_ application in Python

A simple python application running on the host machine can also connect
to the database.

Before you continue, you must have python and
[redis-py](https://github.com/andymccurdy/redis-py#installation)
(python library for connecting to Redis) configured on the host machine
running the container.

1. In the command-line terminal, create a new file called "redis_test.py"

    [code example]

1. Paste this code into the "redis_test.py" file.

    This application stores a value in key1 in cluster 1, gets that value from
    key1 in cluster 1, and gets the value from key1 in cluster 2.

    [code example]

1. To run the "redis_test.py" application, run:

    [code example]

    If the connection is successful, the output of the application looks like:

    [code example]

import redis
    rp1 = redis.StrictRedis(host='localhost', port=12000, db=0)
    rp2 = redis.StrictRedis(host='localhost', port=12002, db=0)
    print ("set key1 123 in cluster 1")
    print (rp1.set('key1', '123'))
    print ("get key1 cluster 1")
    print (rp1.get('key1'))
    print ("get key1 from cluster 2")
    print (rp2.get('key1'))

set key1 123 in cluster 1
    True
    get key1 cluster 1
    "123"
    get key1 from cluster 2
    "123"


Create an Active-Active geo-replicated database

Summary: How to create an Active-Active database and things to consider when setting it up.

Overview

[Active-Active geo-replicated databases]() (formerly known as CRDBs) give applications write access
to replicas of the dataset in different geographical locations.

The participating Redis Software clusters that host the instances can be distributed in different geographic locations.
Every instance of an Active-Active database can receive write operations, and all operations are [synchronized]() to all instances without conflict.

Steps to create an Active-Active database

1. **Create a service account** - On each participating cluster, create a dedicated user account with the Admin role.
1. **Confirm connectivity** - Confirm network connectivity between the participating clusters.
1. **Create Active-Active database** - Connect to one of your clusters and create a new Active-Active database.
1. **Add participating clusters** - Add the participating clusters to the Active-Active database with the user credentials for the service account.
1. **Verify creation** - Log in to each of the participating clusters and verify your Active-Active database was created on them.
1. **Confirm Active-Active database synchronization** - Test writing to one cluster and reading from a different cluster.

Prerequisites

- Two or more machines with the same version of Redis Software installed
- Network connectivity and cluster FQDN name resolution between all participating clusters
- [Network time service]() listener (ntpd) configured and running on each node in all clusters

Create an Active-Active database

1. Create service accounts on each participating cluster:

    1. In a browser, open the Cluster Manager UI for the participating cluster.

        The default address is: `https://<RS_address>:8443`

    1. Go to the **Access Control > Users** tab:

    1. Click **+ Add user**.

    1. Enter the username, email, and password for the user.

    1. Select the **Admin** role.

    1. Click **Save**.

1. To verify network connectivity between participating clusters,
    run the following `telnet` command from each participating cluster to all other participating clusters:

    [code example]

1. In a browser, open the Cluster Manager UI of the cluster where you want to create the Active-Active database.

    The default address is: `https://<RS_address>:8443`

1. Open the **Create database** menu with one of the following methods:

    - Click the **+** button next to **Databases** in the navigation menu:

    - Go to the **Databases** screen and select **Create database**:

1. Select **Active-Active database**.

1. Enter the cluster's local admin credentials, then click **Save**:

1. Add participating clusters that will host instances of the Active-Active database:

    1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

    1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

    1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Enter a **Database name**.

1. If your cluster supports [Redis Flex or Auto Tiering](), in **Runs on** you can select **Flash** so that your database uses Flash memory. We recommend that you use AOF every 1 sec for the best performance during the initial Active-Active database sync of a new replica.

1. To configure additional database settings, expand each relevant section to make changes.

    See [Configuration settings](#configuration-settings) for more information about each setting.

1. Click **Create**.

Configuration settings

- **Database version** - The Redis version used by your database.

- **Database name** - The database name requirements are:

    - Maximum of 63 characters

    - Only letters, numbers, or hyphens (-) are valid characters

    - Must start and end with a letter or digit

    - Case-sensitive

- **Port** - You can define the port number that clients use to connect to the database. Otherwise, a port is randomly selected.

You cannot change the [port number]()
after the database is created.

- **Memory limit** - [Database memory limits]() include all database replicas and shards, including replica shards in database replication and database shards in database clustering.

    If the total size of the database in the cluster reaches the memory limit, the data eviction policy for the database is enforced.

If you create a database with Auto Tiering enabled, you also need to set the RAM-to-Flash ratio
for this database. Minimum RAM is 10%. Maximum RAM is 50%.

- **Memory eviction** - The default [eviction policy]() for Active-Active databases is `noeviction`. Redis Software versions 6.0.20 and later support all eviction policies for Active-Active databases, unless [Redis Flex or Auto Tiering]() is enabled.

- [**Capabilities**]() (previously **Modules**) - When you create a new in-memory database, you can enable additional capabilities in the database. You cannot enable them after database creation.

    Active-Active databases created with or upgraded to Redis version 8 or later automatically enable [search and query]() and [JSON](), which allows you to index, query, and perform full-text searches of nested JSON documents.

    For Auto Tiering databases, you can enable capabilities that support Auto Tiering. See [Redis Software and Redis Stack feature compatibility
]() for compatibility details.

    To add capabilities to the database:

    1. In the **Capabilities** section, select one or more capabilities.

    1. To customize capabilities, select **Parameters** and enter the optional custom configuration.

    1. Select **Done**.

TLS

If you enable TLS when you create the Active-Active database, the nodes use the TLS mode **Require TLS for CRDB communication only** to require TLS authentication and encryption for communications between participating clusters.

After you create the Active-Active database, you can set the TLS mode to **Require TLS for all communications** so client communication from applications are also authenticated and encryption.

High availability

- [**Replication**]() - We recommend that all Active-Active database use replication for best intercluster synchronization performance.

    When replication is enabled, every Active-Active database master shard is replicated to a corresponding replica shard. The replica shards are then used to synchronize data between the instances, and the master shards are dedicated to handling client requests.

- [**Replica high availability**]() - We also recommend that you enable replica high availability to ensure replica shards are highly-available for this synchronization.

Clustering

- In the [**Clustering**]() section, you can either:

    - **Enable sharding** and select the number of shards you want to have in the database. When database clustering is enabled, databases have limitations for [multi-key operations]().

        You can increase the number of shards in the database at any time.

    - Clear the **Enable sharding** option to use only one shard, which allows you to use [multi-key operations]() without the limitations.

You cannot enable or turn off database clustering after the Active-Active database is created.

- [**OSS Cluster API**]() - The OSS Cluster API configuration allows access to multiple endpoints for increased throughput. The OSS Cluster API setting applies to all instances of the Active-Active database across participating clusters.

    This configuration requires clients to connect to the primary node to retrieve the cluster topology before they can connect directly to proxies on each node.

    When you enable the OSS Cluster API, shard placement changes to _Sparse_, and the database proxy policy changes to _All primary shards_ automatically.

Durability

To protect against loss of data stored in RAM, you can enable [**Persistence**]() to store a copy of the data on disk.

Active-Active databases support append-only file (AOF) persistence only. Snapshot persistence is not supported for Active-Active databases.

Access control

- **Unauthenticated access** - You can access the database as the default user without providing credentials.

- **Password-only authentication** - When you configure a password for your database's default user, all connections to the database must authenticate with the [AUTH command](.

    If you also configure an access control list, connections can specify other users for authentication, and requests are allowed according to the Redis ACLs specified for that user.

    Creating a database without ACLs enables a *default* user with full access to the database. You can secure default user access by requiring a password.

- **Access Control List** - You can specify the [user roles]() that have access to the database and the [Redis ACLs]() that apply to those connections.

    You can only configure access control after the Active-Active database is created. In each participating cluster, add ACLs after database creation.

    To define an access control list for a database:

    1. In **Security > Access Control > Access Control List**, select **+ Add ACL**.

    1. Select a [role]() to grant database access.

    1. Associate a [Redis ACL]() with the role and database.

    1. Select the check mark to add the ACL.

Causal consistency

[**Causal consistency**]() in an Active-Active database guarantees that the order of operations on a specific key is maintained across all instances of an Active-Active database.

To enable causal consistency for an existing Active-Active database, use the REST API.

<!-- Also in getting-started-crdbs.md -->

Test Active-Active database connections

With the Redis database created, you are ready to connect to your database. See [Connect to Active-Active databases]() for tutorials and examples of multiple connection methods.


Delete Active-Active databases

Summary: Considerations while deleting Active-Active databases.

Content

When you delete an Active-Active database (formerly known as CRDB),
all instances of the Active-Active database are deleted from all participating clusters.

This action is immediate, non-reversible, and has no rollback.

Because Active-Active databases are made up of instances on multiple participating clusters,
to restore a deleted Active-Active database you must create the database again with all of its instances
and then restore the data to the database from backup.

We recommended that you:

- Back up your data and test the restore on another database before you delete an Active-Active database.
- Consider [flushing the data]() from the database
    so that you can keep the Active-Active database configuration and restore the data to it if necessary.


Application failover with Active-Active databases

Summary: How to failover your application to connect to a remote replica.

Overview

Active-Active Redis deployments don't have a built-in failover or failback mechanism for application connections.
An application deployed with an Active-Active database connects to a replica of the database that is geographically nearby.
If that replica is not available, the application can failover to a remote replica, and failback again if necessary.
In this article we explain how this process works.

For other disaster recovery strategies including network-based, proxy-based, and client library approaches, see [Active-Active disaster recovery strategies]().

Active-Active connection failover can improve data availability, but can negatively impact data consistency.
Active-Active replication, like Redis replication, is asynchronous.
An application that fails over to another replica can miss write operations.
If the failed replica saved the write operations in persistent storage,
then the write operations are processed when the failed replica recovers.

Detecting Failure

Your application can detect two types of failure:

1. **Local failures** - The local replica is down or otherwise unavailable
1. **Replication failures** - The local replica is available but fails to replicate to or from remote replicas

You can also use [database availability API requests]() to determine if a database replica is available to handle read and write operations. The lag-aware database availability requests considers CRDT replication lag as a health check criterion to prevent reading stale data during failback scenarios.

Local Failures

Local failure is detected when the application is unable to connect to the database endpoint for any reason. Reasons for a local failure can include: multiple node failures, configuration errors, connection refused, connection timed out, unexpected protocol level errors.

Replication Failures

Replication failures are more difficult to detect reliably without causing false positives. Replication failures can include: network split, replication configuration issues, remote replica failures.

The most reliable method for health-checking replication is by using the Redis publish/subscribe (pub/sub) mechanism.

Note that this document does not suggest that Redis pub/sub is reliable in the common sense. Messages can get lost in certain conditions, but that is acceptable in this case because typically the application determines that replication is down only after not being able to deliver a number of messages over a period of time.

When you use the pub/sub data type to detect failures, the application:

1. Connects to all replicas and subscribes to a dedicated channel for each replica.
1. Connects to all replicas and periodically publishes a uniquely identifiable message.
1. Monitors received messages and ensures that it is able to receive its own messages within a predetermined window of time.

You can also use known dataset changes to monitor the reliability of the replication stream,
but pub/sub is preferred method because:

1. It does not involve dataset changes.
1. It does not make any assumptions about the dataset.
1. Pub/sub messages are delivered as replicated effects and are a more reliable indicator of a live replication link. In certain cases, dataset keys may appear to be modified even if the replication link fails. This happens because keys may receive updates through full-state replication (re-sync) or through online replication of effects.

Impact of sharding on failure detection

If your sharding configuration is symmetric, make sure to use at least one key (PUB/SUB channels or real dataset key) per shard. Shards are replicated individually and are vulnerable to failure. Symmetric sharding configurations have the same number of shards and hash slots for all replicas.
We do not recommend an asymmetric sharding configuration, which requires at least one key per hash slot that intersects with a pair of shards.

To make sure that there is at least one key per shard, the application should:

1. Use the Cluster API to retrieve the database sharding configuration.
1. Compute a number of key names, such that there is one key per shard.
1. Use those key names as channel names for the pub/sub mechanism.

Failing over

When the application needs to failover to another replica, it should simply re-establish its connections with the endpoint on the remote replica. Because Active/Active and Redis replication are asynchronous, the remote endpoint may not have all of the locally performed and acknowledged writes.

It's best if your application doesn't read its own recent writes. Those writes can be either:

1. Lost forever, if the local replica has an event such as a double failure or loss of persistent files.
1. Temporarily unavailable, but will be available at a later time if the local replica's failure is temporary.

<!---
Sample code that maps a hash slot to a key name can be found in this Python script.
 --->

Failback decision

Your application can use the same checks described above to continue monitoring the state of the failed replica after failover.

To monitor the state of a replica during the failback process, you must make sure the replica is available, re-synced with the remote replicas, and not in stale mode. The PUB/SUB mechanism is an effective way to monitor this.

Dataset-based mechanisms are potentially less reliable for several reasons:
1. In order to determine that a local replica is not stale, it is not enough to simply read keys from it. You must also attempt to write to it.
1. As stated above, remote writes for some keys appear in the local replica before the replication link is back up and while the replica is still in stale mode.
1. A replica that was never written to never becomes stale, so on startup it is immediately ready but serves stale data for a longer period of time.

Replica Configuration Changes

All failover and failback operations should be done strictly on the application side, and should not involve changes to the Active-Active configuration.
The only valid case for re-configuring the Active-Active deployment and removing a replica is when memory consumption becomes too high because garbage collection cannot be performed.
Once a replica is removed, it can only be re-joined as a new replica and it loses any writes that were not converged.


Hashes in an Active-Active databases

Summary: Information about using hashes with an Active-Active database.

Content

Hashes are great for structured data that contain a map of fields and
values. They are used for managing distributed user or app session
state, user preferences, form data and so on. Hash fields contain string
type and string types operate just like the standard Redis string types
when it comes to CRDTs. Fields in hashes can be initialized as a string
using HSET or HMSET or can be used to initialize counter types that are
numeric integers using HINCRBY or floats using HINCRBYFLOAT.

Hashes in Active-Active databases behave the same and maintain additional metadata to
achieve an "OR-Set" behavior to handle concurrent conflicting writes.
With the OR-Set behavior, writes to add new fields across multiple Active-Active database
instances are typically unioned except in cases of conflicts.
Conflicting instance writes can happen when an Active-Active database instance deletes a
field while the other adds the same field. In this case and observed
remove rule is followed. That is, remove can only remove fields it has
already seen and in all other cases element add/update wins.

Field values behave just like CRDT strings. String values can be types
string, counter integer based on the command used for initialization of
the field value. See "String Data Type in Active-Active databases" and "String Data Type
with Counter Value in Active-Active databases" for more details.

Here is an example of an "add wins" case:

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | HSET key1 field1 "a" |  |
|  t2 |  | HSET key1 field2 "b" |
|  t4 | - Sync - | - Sync - |
|  t5 | HGETALL key1
1) "field2"
2) "b"
3) "field1"
4) "a" | HGETALL key1
1) "field2"
2) "b"
3) "field1"
4) "a" |


HyperLogLog in Active-Active databases

Summary: Information about using hyperloglog with an Active-Active database.

Overview

**HyperLogLog** is an algorithm that addresses the [count-distinct problem](https://en.wikipedia.org/wiki/Count-distinct_problem).
To do this it approximates the numbers of items in a [set](https://en.wikipedia.org/wiki/Multiset).
Determining the _exact_ cardinality of a set requires memory according to the cardinality of the set.
Because it estimates the cardinality by probability, the HyperLogLog algorithm can run with more reasonable memory requirements.

HyperLogLog in Redis

Redis Open source implements [HyperLogLog](https://redislabs.com/redis-best-practices/counting/hyperloglog/) (HLL) as a native data structure.
It supports adding elements ([PFADD]() to an HLL, counting elements ([PFCOUNT]() of HLLs, and merging of ([PFMERGE]() HLLs.

Here is an example of a simple write case:

| Time | Replica 1         | Replica 2         |
| ---- | ----------------- | ----------------- |
| t1   | PFADD hll x       |                   |
| t2   | --- sync ---      |                   |
| t3   |                   | PFADD hll y       |
| t4   | --- sync ---      |                   |
| t5   | PFCOUNT hll --> 2 | PFCOUNT hll --> 2 |

Here is an example of a concurrent add case:

| Time | Replica 1         | Replica 2         |
| ---- | ----------------- | ----------------- |
| t1   | PFADD hll x       | PFADD hll y       |
| t2   | PFCOUNT hll --> 1 | PFCOUNT hll --> 1 |
| t3   | --- sync ---      |                   |
| t4   | PFCOUNT hll --> 2 | PFCOUNT hll --> 2 |

The DEL-wins approach

Other collections in the Redis-CRDT implementation use the observed remove method to resolve conflicts.
The CRDT-HLL uses the DEL-wins method.
If a DEL request is received at the same time as any other request (ADD/MERGE/EXPIRE) on the HLL-key
the replicas consistently converge to delete key.
In the observed remove method used by other collections (sets, lists, sorted-sets and hashes),
only the replica that received the DEL request removes the elements, but elements added concurrently in other replicas exist in the consistently converged collection.
We chose to use the DEL-wins method for the CRDT-HLL to maintain the original time and space complexity of the HLL in Redis Open source.

Here is an example of a DEL-wins case:

| HLL  |                 |                 | \|  | Set  |                     |                     |
| ---- | --------------- | --------------- | --- | ---- | ------------------- | ------------------- |
|      |                 |                 | \|  |      |                     |                     |
| Time | Replica 1       | Replica 2       | \|  | Time | Replica 1           | Replica 2           |
|      |                 |                 | \|  |      |                     |                     |
| t1   | PFADD h e1      |                 | \|  | t1   | SADD s e1           |                     |
| t2   | --- sync ---    |                 | \|  | t2   | --- sync ---        |                     |
| t3   | PFCOUNT h --> 1 | PFCOUNT h --> 1 | \|  | t3   | SCARD s --> 1       | SCARD s --> 1       |
| t4   | PFADD h e2      | Del h           | \|  | t4   | SADD s e2           | Del S               |
| t5   | PFCOUNT h --> 2 | PFCOUNT h --> 0 | \|  | t5   | SCARD s --> 2       | SCARD s --> 0       |
| t6   | --- sync ---    |                 | \|  | t6   | --- sync ---        |                     |
| t7   | PFCOUNT h --> 0 | PFCOUNT h --> 0 | \|  | t7   | SCARD s --> 1       | SCARD s --> 1       |
| t8   | Exists h --> 0  | Exists h --> 0  | \|  | t8   | Exists s --> 1      | Exists s --> 1      |
|      |                 |                 | \|  | t9   | SMEMBERS s --> {e2} | SMEMBERS s --> {e2} |

HLL in Active-Active databases versus HLL in Redis Open source

In Active-Active databases, we implemented HLL within the CRDT on the basis of the Redis implementation with a few exceptions:

- Redis keeps the HLL data structure as an encoded string object
    such that you can potentially run any string request can on a key that contains an HLL. In CRDT, only get and set are supported for HLL.
- In CRDT, if you do SET on a key that contains a value encoded as an HLL, then the value will remain an HLL. If the value is not encoded as HLL, then it will be a register.


Data types for Active-Active databases

Summary: Introduction to differences in data types between standalone and Active-Active Redis databases.

Hashes in an Active-Active databases

HyperLogLog in Active-Active databases

Summary: Information about using JSON with an Active-Active database.

JSON in Active-Active databases

Summary: Information about using list with an Active-Active database.

Lists in Active-Active databases

Summary: Information about using sets with an Active-Active database.

Sets in Active-Active databases

Summary: Information about using sorted sets with an Active-Active database.

Sorted sets in Active-Active databases

Summary: Information about using streams with an Active-Active database.

Streams in Active-Active databases

Summary: Information about using strings and bitfields with an Active-Active database.

Strings and bitfields in Active-Active databases

Content

Active-Active databases use conflict-free replicated data types (CRDTs). From a developer perspective, most supported data types work the same for Active-Active and standard Redis databases. However, a few methods also come with specific requirements in Active-Active databases.

Even though they look identical to standard Redis data types, there are specific rules that govern the handling of
conflicting concurrent writes for each data type.

As conflict handling rules differ between data types, some commands have slightly different requirements in Active-Active databases versus standard Redis databases.

See the following articles for more information.


JSON in Active-Active databases

Overview

Active-Active databases support JSON data structures.

The design is based on [A Conflict-Free Replicated JSON Datatype](https://arxiv.org/abs/1608.03960) by Kleppmann and Beresford, but the implementation includes some changes. Several [conflict resolution rule](#conflict-resolution-rules) examples were adapted from this paper as well.

Prerequisites

To use JSON in an Active-Active database, you must enable JSON during database creation.

Active-Active Redis Cloud databases add JSON by default. See [Create an Active-Active subscription]() in the Redis Cloud documentation for details.

In Redis Software, Active-Active databases created with or upgraded to Redis version 8 or later automatically enable JSON. For earlier Redis versions, see [Create an Active-Active JSON database]() in the Redis Stack and Redis Software documentation for instructions.


Lists in Active-Active databases

Content

Redis lists are simply lists of strings, sorted by insertion order. It
is possible to add elements to a Redis List that push new elements to
the head (on the left) or to the tail (on the right) of the list. Redis
lists can be used to easily implement queues (using LPUSH and RPOP, for
example) and stacks (using LPUSH and LPOP, for
example).

Lists in Active-Active databases are just the same as regular Redis Lists. See the
following examples to get familiar with Lists' behavior in an
Active-Active database.

Simple Lists
example:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH mylist "hello" |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | LPUSH mylist "world" |
|  t4 | — Sync — | — Sync — |
|  t5 | LRANGE mylist 0 -1 =>"world" "hello" | LRANGE mylist 0 -1 => "world" "hello" |

**Explanation**:
The final list contains both the "world" and "hello" elements, in that
order (Instance 2 observed "hello" when it added
"world").

Example of Lists with Concurrent
Insertions:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | LINSERT L AFTER x y1 |  |
|  t4 |  | LINSERT L AFTER x y2 |
|  t5 | LRANGE L 0 -1 => x y1 | LRANGE L 0 -1 => x y2 |
|  t6 | — Sync — | — Sync — |
|  t7 | LRANGE L 0 -1 => x y1 y2 | LRANGE L 0 -1 => x y1 y2 |

**Explanation**:
Instance 1 added an element y1 after x, and then Instance 2 added element y2 after x.
The final List contains all three elements: x is the first element, after it y1 and then y2.
The Active-Active database resolves the conflict arbitrarily but applies the resolution consistently across all Active-Active database instances.

Example of Deleting a List while Pushing a New
Element:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | LRANGE L 0 -1 => x | LRANGE L 0 -1 => x |
|  t4 | LPUSH L y | DEL L |
|  t5 | — Sync — | — Sync — |
|  t6 | LRANGE L 0 -1 => y | LRANGE L 0 -1 => y |

**Explanation**
At t4 - t6, DEL deletes only observed elements. This is why L still
contains y.

Example of Popping Elements from a
List:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x y z |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | RPOP L => x |
|  t4 | — Sync — | — Sync — |
|  t5 | RPOP L => y |  |
|  t6 | — Sync — | — Sync — |
|  t7 | RPOP L => z | RPOP L => z |

**Explanation**:
At t1, the operation pushes elements x, y, z to List L. At t3, the
sequential pops behave as expected from a queue. At t7, the concurrent
pop in both instances might show the same result. The instance was not
able to sync regarding the z removal so, from the point of view of each
instance, z is located in the List and can be popped. After syncing,
both lists are empty.

Be aware of the behavior of Lists in Active-Active databases when using List as a stack
or queue. As seen in the above example, two parallel RPOP operations
performed by two different Active-Active database instances can get the same element in
the case of a concurrent operation. Lists in Active-Active databases guarantee that each
element is POP-ed at least once, but cannot guarantee that each
element is POP-ed only once. Such behavior should be taken into
account when, for example, using Lists in Active-Active databases as building blocks for
inter-process communication systems.

In that case, if the same element cannot be handled twice by the
applications, it's recommended that the POP operations be performed by
one Active-Active database instance, whereas the PUSH operations can be performed by
multiple instances.


Sets in Active-Active databases

Content

A Redis set is an unordered collection of strings. It is possible to
add, remove, and test for the existence of members with Redis commands.
A Redis set maintains a unique collection of elements. Sets can be great
for maintaining a list of events (click streams), users (in a group
conversation), products (in recommendation lists), engagements (likes,
shares) and so on.

Sets in Active-Active databases behave the same and maintain additional metadata to
achieve an "OR-Set" behavior to handle concurrent conflicting
writes. With the OR-Set behavior, writes across multiple Active-Active database instances
are typically unioned except in cases of conflicts. Conflicting instance
writes can happen when a Active-Active database instance deletes an element while the
other adds the same element. In this case and observed remove rule is
followed. That is, remove can only remove instances it has already seen
and in all other cases element add wins.

Here is an example of an "add wins" case:

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | SADD key1 "a" |  |
|  t2 |  | SADD key1 "b" |
|  t3 | SMEMBERS key1 "a" | SMEMBERS key1 "b" |
|  t4 | — Sync — | — Sync — |
|  t3 | SMEMBERS key1 "a" "b" | SMEMBERS key1 "a" "b" |

Here is an example of an "observed remove" case.

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | SMEMBERS key1 "a" "b" | SMEMBERS key1 "a" "b" |
|  t2 | SREM key1 "a" | SADD key1 "c" |
|  t3 | SREM key1 "c" |  |
|  t4 | — Sync — | — Sync — |
|  t3 | SMEMBERS key1 "c" "b" | SMEMBERS key1 "c" "b" |


Sorted sets in Active-Active databases

Content

[Redis Geospatial (Geo)]() is based on Sorted Sets, so the same Active-Active database development instructions apply to Geo.

Similar to Redis Sets, Redis Sorted Sets are non-repeating collections
of Strings. The difference between the two is that every member of a
Sorted Set is associated with a score used to order the Sorted Set from
lowest to highest. While members are unique, they may have the same
score.

With Sorted Sets, you can quickly add, remove or update elements as
well as get ranges by score or by rank (position). Sorted Sets in Active-Active databases
behave the same and maintain additional metadata to handle concurrent
conflicting writes. Conflict resolution is done in two
phases:

1. First, the database resolves conflict at the set level using "OR
    Set" (Observed-Remove Set). With OR-Set behavior, writes across
    multiple Active-Active database instances are typically unioned except in cases of
    conflicts. Conflicting writes can happen when an Active-Active database instance
    deletes an element while the other adds or updates the same element.
    In this case, an observed Remove rule is followed, and only
    instances it has already seen are removed. In all other cases, the
    Add / Update element wins.
1. Second, the database resolves conflict at the score level. In this
    case, the score is treated as a counter and applies the same
    conflict resolution as regular counters.

See the following examples to get familiar with Sorted Sets'
behavior in Active-Active database:

Example of Simple Sorted Set with No
Conflict:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | ZADD Z 1.2 y |
|  t4 | — Sync — | — Sync — |
|  t5 | ZRANGE Z 0 -1 => x y | ZRANGE Z 0 -1 => x y |

**Explanation**:
When adding two different elements to a Sorted Set from different
replicas (in this example, x with score 1.1 was added by Instance 1 to
Sorted Set Z, and y with score 1.2 was added by Instance 2 to Sorted Set
Z) in a non-concurrent manner (i.e. each operation happened separately
and after both instances were in sync), the end result is a Sorted
Set including both elements in each Active-Active database instance.
Example of Sorted Set and Concurrent
Add:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 |  | ZADD Z 2.1 x |
|  t3 | ZSCORE Z x => 1.1 | ZSCORE Z x => 2.1 |
|  t4 | — Sync — | — Sync — |
|  t5 | ZSCORE Z x => 2.1 | ZSCORE Z x => 2.1 |

**Explanation**:
When concurrently adding an element x to a Sorted Set Z by two different
Active-Active database instances (Instance 1 added score 1.1 and Instance 2 added score
2.1), the Active-Active database implements Last Write Win (LWW) to determine the score of
x. In this scenario, Instance 2 performed the ZADD operation at time
t2\>t1 and therefore the Active-Active database sets the score 2.1 to
x.

Example of Sorted Set with Concurrent Add Happening at the Exact Same
Time:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x | ZADD Z 2.1 x |
|  t2 | ZSCORE Z x => 1.1 | ZSCORE Z x => 2.1 |
|  t3 | — Sync — | — Sync — |
|  t4 | ZSCORE Z x => 1.1 | ZSCORE Z x => 1.1 |

**Explanation**:
The example above shows a relatively rare situation, in which two Active-Active database
instances concurrently added the same element x to a Sorted Set at the
same exact time but with a different score, i.e. Instance 1 added x with
a 1.1 score and Instance 2 added x with a 2.1 score. After syncing, the
Active-Active database realized that both operations happened at the same time and
resolved the conflict by arbitrarily (but consistently across all Active-Active database
instances) giving precedence to Instance 1.
Example of Sorted Set with Concurrent Counter
Increment:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 1.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | ZINCRBY Z 1.0 x | ZINCRBY Z 1.0 x |
|  t4 | — Sync — | — Sync — |
|  t5 | ZSCORE Z x => 3.1 | ZSCORE Z x => 3.1 |

**Explanation**:
The result is the sum of all
ZINCRBY
operations performed by all Active-Active database instances.

Example of Removing an Element from a Sorted
Set:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | ZADD Z 4.1 x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | ZSCORE Z x => 4.1 | ZSCORE Z x => 4.1 |
|  t4 | ZREM Z x | ZINCRBY Z 2.0 x |
|  t5 | ZSCORE Z x => nill | ZSCORE Z x => 6.1 |
|  t6 | — Sync — | — Sync — |
|  t7 | ZSCORE Z x => 2.0 | ZSCORE Z x => 2.0 |

**Explanation**:
At t4 - t5, concurrent ZREM and ZINCRBY operations ran on Instance 1
and Instance 2 respectively. Before the instances were in sync, the ZREM
operation could only delete what had been seen by Instance 1, so
Instance 2 was not affected. Therefore, the ZSCORE operation shows the
local effect on x. At t7, after both instances were in-sync, the Active-Active database
resolved the conflict by subtracting 4.1 (the value of element x in
Instance 1) from 6.1 (the value of element x in Instance 2).


Streams in Active-Active databases

Overview

A [Redis Stream]() is a data structure that acts like an append-only log.
Each stream entry consists of:

- A unique, monotonically increasing ID
- A payload consisting of a series key-value pairs

You add entries to a stream with the XADD command. You access stream entries using the XRANGE, XREADGROUP, and XREAD commands (however, see the caveat about XREAD below).

Streams and Active-Active

Active-Active databases allow you to write to the same logical stream from more than one region.
Streams are synchronized across the regions of an Active-Active database.

In the example below, we write to a stream concurrently from two regions. Notice that after syncing, both regions have identical streams:

Time
Region 1
Region 2

t1
XADD messages * text hello
XADD messages * text goodbye

t2
XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929246795-2]

t3
— Sync —
— Sync —

t4
XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]
XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]

Notice also that the synchronized streams contain no duplicate IDs. As long as you allow the database to generate your stream IDs, you'll never have more than one stream entry with the same ID.

Redis Open Source uses one radix tree (referred to as `rax` in the code base) to implement each stream. However, Active-Active databases implement a single logical stream using one `rax` per region.
Each region adds entries only to its associated `rax` (but can remove entries from all `rax` trees).
This means that XREAD and XREADGROUP iterate simultaneously over all `rax` trees and return the appropriate entry by comparing the entry IDs from each `rax`.

Conflict resolution

Active-Active databases use an "observed-remove" approach to automatically resolve potential conflicts.

With this approach, a delete only affects the locally observable data.

In the example below, a stream, `x`, is created at _t1_. At _t3_, the stream exists in two regions.

Time
Region 1
Region 2

t1
XADD messages * text hello

t2
— Sync —
— Sync —

t3
XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929244828-1]

t4
DEL messages
XADD messages * text goodbye

t5
— Sync —
— Sync —

t6
XRANGE messages - +
→ [1589929246795-2]
XRANGE messages - +
→ [1589929246795-2]

At _t4_, the stream is deleted from Region 1. At the same time, an entry with ID ending in `3700` is added to the same stream at Region 2. After the sync, at _t6_, the entry with ID ending in `3700` exists in both regions. This is because that entry was not visible when the local stream was deleted at _t4_.

ID generation modes

Usually, you should allow Redis streams generate its own stream entry IDs. You do this by specifying `*` as the ID in calls to XADD. However, you _can_ provide your own custom ID when adding entries to a stream.

Because Active-Active databases replicate asynchronously, providing your own IDs can create streams with duplicate IDs. This can occur when you write to the same stream from multiple regions.

| Time | Region 1                      | Region 2                      |
| ---- | ------------------------------- | ------------------------------- |
| _t1_   | `XADD x 100-1 f1 v1`              | `XADD x 100-1 f1 v1`              |
| _t2_   | _— Sync —_                    | _— Sync —_                    |
| _t3_   | `XRANGE x - +`
 **→ [100-1, 100-1]** | `XRANGE x - +`
 **→ [100-1, 100-1]** |

In this scenario, two entries with the ID `100-1` are added at _t1_. After syncing, the stream `x` contains two entries with the same ID.

Stream IDs in Redis Open Source consist of two integers separated by a dash ('-'). When the server generates the ID, the first integer is the current time in milliseconds, and the second integer is a sequence number. So, the format for stream IDs is MS-SEQ.

To prevent duplicate IDs and to comply with the original Redis streams design, Active-Active databases provide three ID modes for XADD:

1. **Strict**: In _strict_ mode, XADD allows server-generated IDs (using the '`*`' ID specifier) or IDs consisting only of the millisecond (MS) portion. When the millisecond portion of the ID is provided, the ID's sequence number is calculated using the database's region ID. This prevents duplicate IDs in the stream. Strict mode rejects full IDs (that is, IDs containing both milliseconds and a sequence number).
1. **Semi-strict**: _Semi-strict_ mode is just like _strict_ mode except that it allows full IDs (MS-SEQ). Because it allows full IDs, duplicate IDs are possible in this mode.
1. **Liberal**: XADD allows any monotonically ascending ID. When given the millisecond portion of the ID, the sequence number will be set to `0`. This mode may also lead to duplicate IDs.

The default and recommended mode is _strict_, which prevents duplicate IDs.

Why do you want to prevent duplicate IDs? First, XDEL, XCLAIM, and other commands can affect more than one entry when duplicate IDs are present in a stream. Second, duplicate entries may be removed if a database is exported or renamed.

To change XADD's ID generation mode, use the `rladmin` command-line utility:

Set _strict_ mode:
[code example]

Set _semi-strict_ mode:
[code example]

Set _liberal_ mode:
[code example]

Iterating a stream with XREAD

In Redis Open Source and in non-Active-Active databases, you can use XREAD to iterate over the entries in a Redis Stream. However, with an Active-Active database, XREAD may skip entries. This can happen when multiple regions write to the same stream.

In the example below, XREAD skips entry `115-2`.

| Time | Region 1                                         | Region 2                                         |
| ---- | -------------------------------------------------- | -------------------------------------------------- |
| _t1_   | `XADD x 110 f1 v1`                                   | `XADD x 115 f1 v1`                                   |
| _t2_   | `XADD x 120 f1 v1`                                   |                                                    |
| _t3_   | `XADD x 130 f1 v1`                                   |                                                    |
| _t4_   | `XREAD COUNT 2 STREAMS x 0`
**→ [110-1, 120-1]**       |                                                    |
| _t5_   | _— Sync —_                                       | _— Sync —_                                       |
| _t6_   | `XREAD COUNT 2 STREAMS x 120-1`
**→ [130-1]**            |                                                    |
| _t7_   | `XREAD STREAMS x 0`
**→[110-1, 115-2, 120-1, 130-1]** | `XREAD STREAMS x 0`
**→[110-1, 115-2, 120-1, 130-1]** |

You can use XREAD to reliably consume a stream only if all writes to the stream originate from a single region. Otherwise, you should use XREADGROUP, which always guarantees reliable stream consumption.

Consumer groups

Active-Active databases fully support consumer groups with Redis Streams. Here is an example of creating two consumer groups concurrently:

| Time | Region 1                  | Region 2                  |
| ---- | --------------------------- | --------------------------- |
| _t1_   | `XGROUP CREATE x group1 0`        | `XGROUP CREATE x group2 0`        |
| _t2_   | `XINFO GROUPS x`
**→ [group1]**     | `XINFO GROUPS x`
**→ [group2]**     |
| _t3_   | _— Sync —_                | — Sync —                |
| _t4_   | `XINFO GROUPS x`
**→ [group1, group2]** | `XINFO GROUPS x`
**→ [group1, group2]** |

Redis Open Source uses one radix tree (`rax`) to hold the global pending entries list and another `rax` for each consumer's PEL.
The global PEL is a unification of all consumer PELs, which are disjoint.

An Active-Active database stream maintains a global PEL and a per-consumer PEL for each region.

When given an ID different from the special ">" ID, XREADGROUP iterates simultaneously over all of the PELs for all consumers.
It returns the next entry by comparing entry IDs from the different PELs.

Conflict resolution

The "delete wins" approach is a way to automatically resolve conflicts with consumer groups.
In case of concurrent consumer group operations, a delete will "win" over other concurrent operations on the same group.

In this example, the DEL at _t4_ deletes both the observed `group1` and the non-observed `group2`:

| Time | Region 1              | Region 2              |
| ---- | ----------------------- | ----------------------- |
| _t1_   | `XGROUP CREATE x group1 0`    |                         |
| _t2_   | _— Sync —_            | _— Sync —_            |
| _t3_   | `XINFO GROUPS x`
**→ [group1]** | `XINFO GROUPS x`
**→ [group1]** |
| _t4_   | `DEL x`                   | `XGROUP CREATE x group2 0`    |
| _t5_   | _— Sync —_            | _— Sync —_            |
| _t6_   | `EXISTS x`
**→ 0** | `EXISTS x`
**→  0**          |

In this example, the XGROUP DESTROY at _t4_ affects both the observed `group1` created in Region 1 and the non-observed `group1` created in Region 3:

| time | Region 1              | Region 2              | Region 3            |
| ---- | ----------------------- | ----------------------- | --------------------- |
| _t1_   | `XGROUP CREATE x group1 0`    |                         |                       |
| _t2_   | _— Sync —_            | _— Sync —_            |                       |
| _t3_   | `XINFO GROUPS x`
**→ [group1]** | `XINFO GROUPS x`
**→  [group1]** | `XINFO GROUPS x`
**→  []** |
| _t4_   |                         | `XGROUP DESTROY x group1`     | `XGROUP CREATE x group1 0` |
| _t5_   | _— Sync —_            | _— Sync —            | — Sync —          |
| _t6_   | `EXISTS x`
**→  0**          | `EXISTS x`
**→ 0**          | `EXISTS x`
**→  0**    |

Group replication

Calls to XREADGROUP and XACK change the state of a consumer group or consumer. However, it's not efficient to replicate every change to a consumer or consumer group.

To maintain consumer groups in Active-Active databases with optimal performance:

1. Group existence (CREATE/DESTROY) is replicated.
1. Most XACK operations are replicated.
1. Other operations, such as XGROUP, SETID, DELCONSUMER, are not replicated.

For example:

| Time | Region 1                                        | Region 2               |
| ---- | ------------------------------------------------- | ------------------------ |
| _t1_   | `XADD messages 110 text hello`                                |                          |
| _t2_   | `XGROUP CREATE messages group1 0`                              |                          |
| _t3_   | `XREADGROUP GROUP group1 Alice STREAMS messages >`
**→ [110-1]** |                          |
| _t4_   | _— Sync —_                                      | _— Sync —_             |
| _t5_   | `XRANGE messages - +`
**→ [110-1]**                          | XRANGE messages - +
**→ [110-1]** |
| _t6_   | `XINFO GROUPS messages`
**→ [group1]**                            | XINFO GROUPS messages
**→ [group1]**   |
| _t7_   | `XINFO CONSUMERS messages group1`
**→ [Alice]**                      | XINFO CONSUMERS messages group1
**→ []**  |
| _t8_   | `XPENDING messages group1 - + 1`
**→ [110-1]**                     | XPENDING messages group1 - + 1
**→ []** |

Using XREADGROUP across regions can result in regions reading the same entries.
This is due to the fact that Active-Active Streams is designed for at-least-once reads or a single consumer.
As shown in the previous example, Region 2 is not aware of any consumer group activity, so redirecting the XREADGROUP traffic from Region 1 to Region 2 results in reading entries that have already been read.

Replication performance optimizations

Consumers acknowledge messages using the XACK command. Each ack effectively records the last consumed message. This can result in a lot of cross-region traffic. To reduce this traffic, we replicate XACK messages only when all of the read entries are acknowledged.

| Time | Region 1                                                      | Region 2   | Explanation                                                                                |
| ---- | --------------------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------- |
| _t1_   | `XADD x 110-0 f1 v1`                                              |              |                                                                                |
| _t2_   | `XADD x 120-0 f1 v1`                                              |              |                                                                                |
| _t3_   | `XADD x 130-0 f1 v1`                                              |              |                                                                                |
| _t4_   | `XGROUP CREATE x group1 0`                                            |              |                                                                                |
| _t5_   | `XREADGROUP GROUP group1 Alice STREAMS x >` <br/>**→ [110-0, 120-0, 130-0]** |              |                                                                                |
| _t6_   | `XACK x group1 110-0`                                                   |              |                                                                                |
| _t7_   | _— Sync —_                                                    | _— Sync —_ | 110-0 and its preceding entries (none) were acknowledged. We replicate an XACK effect for 110-0.                |
| _t8_   | `XACK x group1 130-0`                                                   |              |                                                                                |
| _t9_   | _— Sync —_                                                    | _— Sync —_ | 130-0 was acknowledged, but not its preceding entries (120-0). We DO NOT replicate an XACK effect for 130-0     |
| _t10_  | `XACK x group1 120-0`                                                   |              |                                                                                |
| _t11_  | _— Sync —_                                                    | _— Sync —_ | 120-0 and its preceding entries (110-0 through 130-0) were acknowledged. We replicate an XACK effect for 130-0. |

In this scenario, if we redirect the XREADGROUP traffic from Region 1 to Region 2 we do not re-read entries 110-0, 120-0 and 130-0.
This means that the XREADGROUP does not return already-acknowledged entries.

Guarantees

Unlike XREAD, XREADGOUP will never skip stream entries.

In traffic redirection, XREADGROUP may return entries that have been read but not acknowledged. It may also even return entries that have already been acknowledged.

When entries are added from multiple regions, Redis guarantees ordered entries in the reply of a single read command (`XREAD`, `XREADGROUP`, or `XRANGE`). However, ordered IDs are not guaranteed for a sequence of read commands.
For example, two subsequent `XREADGROUP GROUP group consumer COUNT 1 STREAMS key >` commands may reply with decreasing IDs.

Summary

With Active-Active streams, you can write to the same logical stream from multiple regions. As a result, the behavior of Active-Active streams differs somewhat from the behavior you get with Redis Open Source. This is summarized below:

Stream commands

1. When using the _strict_ ID generation mode, XADD does not permit full stream entry IDs (that is, an ID containing both MS and SEQ).
1. XREAD may skip entries when iterating a stream that is concurrently written to from more than one region. For reliable stream iteration, use XREADGROUP instead.
1. XSETID fails when the new ID is less than current ID.

Consumer group notes

The following consumer group operations are replicated:

1. Consecutive XACK operations
1. Consumer group creation and deletion (that is, XGROUP CREATE and XGROUP DESTROY)

All other consumer group metadata is not replicated.

A few other notes:

1. XGROUP SETID and DELCONSUMER are not replicated.
1. Consumers exist locally (XREADGROUP creates a consumer implicitly).
1. Renaming a stream (using RENAME) deletes all consumer group information.

rladmin tune db crdb crdt_xadd_id_uniqueness_mode strict

rladmin tune db crdb crdt_xadd_id_uniqueness_mode semi-strict

rladmin tune db crdb crdt_xadd_id_uniqueness_mode liberal


Overview

Active-Active databases support both strings and bitfields.

Active-Active **bitfield** support was added in RS version 6.0.20.

Changes to both of these data structures will be replicated across Active-Active member databases.

Replication semantics

Except in the case of [string counters]() (see below), both strings and bitfields are replicated using a "last write wins" approach. The reason for this is that strings and bitfields are effectively binary objects. So, unlike with lists, sets, and hashes, the conflict resolution semantics of a given operation on a string or bitfield are undefined.

How "last write wins" works

A wall-clock timestamp (OS time) is stored in the metadata of every string
and bitfield operation. If the replication syncer cannot determine the order of operations,
the value with the latest timestamp wins. This is the only case with Active-Active databases where OS time is used to resolve a conflict.

Here's an example where an update happening to the same key at a later
time (t2) wins over the update at t1.

|  **Time** | **Region 1** | **Region 2** |
|  :------: | :------: | :------: |
|  t1 | SET text "a" |  |
|  t2 |  | SET text "b" |
|  t3 | — Sync — | — Sync — |
|  t4 | SET text "c" |  |
|  t5 | — Sync — | — Sync — |
|  t6 |  | SET text "d" |

String counter support

When you're using a string as counter (for instance, with the [INCR]() or [INCRBY]() commands),
then conflicts will be resolved semantically.

On conflicting writes, counters accumulate the total counter operations
across all member Active-Active databases in each sync.

Here's an example of how counter
values works when synced between two member Active-Active databases. With
each sync, the counter value accumulates the private increment and
decrements of each site and maintain an accurate counter across
concurrent writes.

|  **Time** | **Region 1** | **Region 2** |
|  :------: | :------: | :------: |
|  t1 | INCRBY counter 7 |  |
|  t2 |  | INCRBY counter 3 |
|  t3 | GET counter
7 | GET counter
3 |
|  t4 | — Sync — | — Sync — |
|  t5 | GET counter
10 | GET counter
10 |
|  t6 | DECRBY counter 3 |  |
|  t7 |  | INCRBY counter 6 |
|  t8 | — Sync — | — Sync — |
|  t9 | GET counter
13 | GET counter
13 |

Active-Active databases support 59-bit counters.
This limitation is to protect from overflowing a counter in a concurrent operation.