title: Alerts and events

summary: Logged alerts and events

title: Overview

text: The following alerts and events can appear in `syslog` and the Cluster Manager UI logs.

title: Alerts

text:
| Alert | UI message | Severity | Notes |
|-------|------------|----------|-------|
| aof_slow_disk_io | Redis performance is degraded as a result of disk I/O limits | True: error, False: info | Node alert |
| authentication_err | Error authenticating with the source database | error | BDB event |
| backup_delayed | Periodic backup has been delayed for longer than `<threshold>` minutes | True: warning, False: info | BDB alert; Has threshold parameter in the data section of the log entry |
| cpu_utilization | CPU utilization has reached `<threshold>`% | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| even_node_count | True high availability requires an odd number of nodes | True: warning, False: info | Cluster alert |
| ephemeral_storage | Ephemeral storage has reached `<threshold>`% of its capacity | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| free_flash | Flash storage has reached `<threshold>`% of its capacity | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| high_latency | Latency is higher than `<threshold>` milliseconds | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| high_syncer_lag | Sync lag is higher than `<threshold>` seconds | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| high_throughput | Throughput is higher than `<threshold>` RPS (requests per second) | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| inconsistent_rl_sw | Not all nodes in the cluster are running the same Redis Labs Enterprise Cluster version | True: warning, False: info | Cluster alert |
| insufficient_disk_aofrw | Node has insufficient disk space for AOF rewrite | True: error, False: info | Node alert |
| memory | Node memory has reached `<threshold>`% of its capacity | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| multiple_nodes_down | Multiple cluster nodes are down - this might cause data loss | True: warning, False: info | Cluster alert |
| net_throughput | Network throughput has reached `<threshold>` MB/s | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| ocsp_query_failed | Failed querying OCSP server | True: error, False: info | Cluster alert |
| ocsp_status_revoked | OCSP status revoked | True: error, False: info | Cluster alert |
| persistent_storage | Persistent storage has reached `<threshold>`% of its capacity | True: warning, False: info | Node alert; Has global_threshold parameter in the key/value section of the log entry |
| ram_dataset_overhead | RAM Dataset overhead in a shard has reached `<threshold>`% of its RAM limit | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| ram_overcommit | Cluster capacity is less than total memory allocated to its databases | True: error, False: info | Cluster alert |
| ram_values | Percent of values in a shard's RAM is lower than `<threshold>`% of its key count | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| shard_num_ram_values | Number of values in a shard's RAM is lower than `<threshold>` values | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| size | Dataset size has reached `<threshold>`% of the memory limit | True: warning, False: info | BDB alert; Has threshold parameter in the key/value section of the log entry |
| syncer_connection_error | Syncer connection error | error | BDB alert |
| syncer_general_error | Syncer general error | error | BDB alert |
| too_few_nodes_for_replication | Database replication requires at least two nodes in cluster | True: warning, False: info | Cluster alert |

title: Events

text:
| Event | UI message | Severity | Notes |
|-------|------------|----------|-------|
| backup_failed | Backup failed | error | BDB event |
| backup_started | Backup started | info | BDB event |
| backup_succeeded | Backup succeeded | info | BDB event |
| bdb_created | Database created | info | BDB event |
| bdb_deleted | Database deleted | info | BDB event |
| bdb_updated | Database updated | info | BDB event; Indicates that a BDB configuration has been updated |
| checks_error | Node checks error | error | Node event; Indicates that one or more node checks have failed |
| cluster_updated | Cluster settings updated | info | Cluster event; Indicates that cluster settings have been updated |
| compression_unsup_err | Compression not supported by sync destination | error | BDB event |
| crossslot_err | Sharded destination does not support operation executed on source | error | BDB event |
| export_failed | Export failed | error | BDB event |
| export_started | Export started | info | BDB event |
| export_succeeded | Export succeeded | info | BDB event |
| import_failed | Import failed | error | BDB event |
| import_started | Import started | info | BDB event |
| import_succeeded | Import succeeded | info | BDB event |
| license_added | License added | info | Cluster event |
| license_deleted | License deleted | info | Cluster event |
| license_updated | License updated | info | Cluster event |
| node_abort_remove_request | Node abort remove request | info | Node event |
| node_joined | Node joined | info | Cluster event |
| node_operation_failed | Node operation failed | error | Cluster event |
| node_remove_abort_completed | Node remove abort completed | info | Cluster event; Node remove is a process that can fail and can also be aborted. If aborted, the abort can succeed or fail |
| node_remove_abort_failed | Node remove abort failed | error | Cluster event; Node remove is a process that can fail and can also be aborted. If aborted, the abort can succeed or fail |
| node_remove_completed | Node removed | info | Cluster event; Node remove is a process that can fail and can also be aborted. If aborted, the abort can succeed or fail |
| node_remove_failed | Node removed | error | Cluster event; Node remove is a process that can fail and can also be aborted. If aborted, the abort can succeed or fail |
| node_remove_request | Node remove request | info | Node event |
| user_created | User created | info | User event |
| user_deleted | User deleted | info | User event |
| user_updated | User updated | info | User event; Indicates that a user configuration has been updated |


title: Diagnostic logging

summary: Diagnostic logs can help troubleshoot Redis Software.

page_type: content

title: Overview

role: overview

text: The diagnostic logging service collects detailed system logs, which you can use to troubleshoot Redis Software.

title: View diagnostic logs

role: content

text: Diagnostic logs are collected at scheduled intervals and saved in the `/var/opt/redislabs/log/diagnostics/` directory. Each diagnostic log file is overwritten with the new data at the scheduled collection interval.

title: View log collector settings

text:
To view the current log collection schedule and parameters for each log collector, use the REST API to [get the diagnostic logging service configuration]().

[code example]

Example response:

[code example]

title: Change log collector settings

text:
To change how often a log is collected, set the `cron_expression` when you [update the diagnostic logging service configuration]() with the REST API.

[code example]

- Replace `<log_collector>` with the name of the log collector you want to turn off.

- Use standard [cron syntax](https://en.wikipedia.org/wiki/Cron) to set the collection interval in the `cron_expression`.

For the slowlog collector only, you can also set `max_entries` to change the maximum number of slow log entries to collect:

[code example]

title: Turn off log collectors

text:
To turn off a log collector, set its `cron_expression` to an empty string when you [update the diagnostic logging service configuration]() with the REST API.

[code example]

Replace `<log_collector>` with the name of the log collector you want to turn off.

title: Log collectors

text:
Each log collector runs independently and writes a separate log file. The following table describes the log collectors.

Default `cron_expression` values:

- `*/10 * * * *`: Logs are collected every 10 minutes.

- `*/30 * * * *`: Logs are collected every 30 minutes.

| Log collector | Description |
|---------------|-------------|
| bdb | Logs database metadata similar to [`GET /bdbs`](). Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| bdb_client_list | Logs database client lists, with a separate file for each database. Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| bdb_info | Logs the result of running [`INFO ALL`]() on a database, excluding `commandstats`, with a separate file for each database. Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| command_stats | Logs [`INFO commandstats`]() for each database, with a separate file for each database. Each entry is in JSON format.
Default settings:
"cron_expression": "*/30 * * * *" |
| network_stats | Logs the node's network statistics.
Default settings:
"cron_expression": "*/30 * * * *" |
| persistent_files | Lists persistent files from `/var/opt/redislabs/persist/redis`
Default settings:
"cron_expression": "*/10 * * * *" |
| rladmin_status | Logs data about nodes, databases, endpoints, and shards from [`rladmin status`](). Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| shard | Logs shard status similar to [`GET /shards`](). Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| shard_info | Logs the result of running [INFO ALL]() on a shard, with a separate file for each shard. Each entry is in JSON format.
Default settings:
"cron_expression": "*/10 * * * *" |
| shard_latency | Logs the result of running [`latency latest`]() on a shard, with a separate file for each shard.
Default settings:
"cron_expression": "*/10 * * * *" |
| `shard_latency_histogram` | Logs the result of running [`latency histogram`]() on a shard, with a separate file for each shard.
Default settings:
"cron_expression": "*/10 * * * *" |
| slowlog | Logs slow commands from the databases using [`SLOWLOG GET`](), with a separate file for each database. Each entry is in JSON format. The log is sanitized, only the commands are visible.
Default settings:
"cron_expression": "*/10 * * * *",
"max_entries": 100 |
| socket_files | Lists socket files used by Redis Software.
Default settings:
"cron_expression": "*/10 * * * *" |

code: GET /v1/diagnostics

section_id: view-log-collector-settings

code:
PUT /v1/diagnostics
{
  "<log_collector>_target": {
    "cron_expression": "*/5 * * * *"
  }
}

section_id: change-log-collector-settings

code:
PUT /v1/diagnostics
{
  "slowlog_target": {
    "cron_expression": "*/5 * * * *",
    "max_entries": 200
  }
}

code:
PUT /v1/diagnostics
{
  "<log_collector>_target": {
    "cron_expression": ""
  }
}

section_id: turn-off-log-collectors


title: Logging events

summary: Logged alerts and events

title: Alerts and events

title: Diagnostic logging

title: Manage logs

summary: This document explains the structure of Redis Software log entries in `rsyslog` and how to use these log entries to identify events.

title: rsyslog logging

title: View and manage Redis slow log

title: Overview

text:
Management actions performed with Redis Software are logged to make sure system management tasks are appropriately performed or monitored by administrators and for compliance with regulatory standards.

Log entries contain the
following information:

1. Who performed the action?
1. What exactly was the performed action?
1. When was the action performed?
1. Did the action succeed or not?

To get the list of logged events, you can use the REST API or
the **Logs** screen in the UI. The **Logs** screen displays the system and user
events regarding alerts, notifications, and configuration.

You can use the **Logs** screen to review what actions a user has performed, such as editing a database's configuration.

- [Redis slow
    log]()
- [rsyslog logging]()

title: View logs in the UI

text:
Redis Software provides log files for auditing cluster management actions and troubleshooting. You can view these logs in the UI and on the host operating system.

To view event logs in the new Cluster Manager UI, go to **Cluster > Logs**.

title: View logs on the server

text:
Server logs can be found by default in the directory `/var/opt/redislabs/log/`.

These log files are used by the Redis support team to troubleshoot issues. The logs you will most frequently interact with is 'event_log.log'. This log file is where logs of configuration actions within Redis are stored and is useful to determine events that occur within Redis Software.

title: Configure log timestamps

role: configuration

text:
Redis Software allows you to configure log timestamps. To configure log timestamps in the new Cluster Manager UI:

1. Go to **Cluster > Configuration > General**.

1. Change the **Time zone** for the logs based on your location.


title: Manage logs

title: Overview

text: Redis Software comes with [a set of logs]() on the server and available through the user interface to assist users in investigating actions taken on the server and to troubleshoot issues.

title: Send logs to a remote logging server

text:
Redis Software sends logs to syslog by default. You can send these logs to a remote logging server by configuring syslog.

To do this, modify the syslog or rsyslog configuration on your operating system to send logs in the `$logdir` directory (`/var/opt/redislabs/log` in default installations) to a remote monitoring server of your choice. See [rsyslog logging]() for additional details.

title: Log rotation

text:
Redis Software's job scheduler runs `logrotate` every five minutes to examine logs stored on the operating system and rotate them based on the log rotation configuration. You can find the log rotation configuration file at `$pkgconfdir/logrotate.conf` as of Redis Software version 7.2 (`pkgconfdir` is `/opt/redislabs/config` by default, but can be changed in a custom installation).

By default, log rotation occurs when a log exceeds 200 MB. We recommend sending log files to a remote logging server so you can maintain them more effectively.

The following log rotation policy is enabled by default in Redis Software, but you can modify it as needed.

[code example]

- `${logdir}/*.log`: `logrotate` checks the files under the `$logdir` directory (`/var/opt/redislabs/log/`) and rotates any files that end with the extension `.log`.

- `${logdir}/cluster_wd.log.long_retention`: `cluster_wd.log` is copied to `cluster_wd.log.long_retention` before rotation. This copy is kept longer than usual, which is 30 days by default.

- `maxsize 200M`: Rotate log files that exceed 200 MB.

- `daily`: Rotate logs every day regardless of their size.

- `missingok`: If there are missing log files, do nothing.

- `copytruncate`: Truncate the original log file to zero sizes after creating a copy.

- `rotate 10`: Save a maximum of 10 rotated log files. To keep effectively infinite log files, use `rotate 2000` instead.

- `compress`: gzip log files.

- `maxage 7`: Keep the rotated log files for 7 days.

- `notifempty`: Don't rotate the log file if it is empty.

For large scale deployments, you might need to rotate logs at faster intervals than daily. You can also use a cronjob or external vendor solutions.

code:
${logdir}/*.log {
    su ${osuser} ${osgroup}
    maxsize 200M
    daily
    missingok
    copytruncate
    rotate 10
    maxage 7
    compress
    notifempty
    nodateext
    nosharedscripts
    prerotate
        # copy cluster_wd log to another file that will have longer retention
        if [ "\$1" = "${logdir}/cluster_wd.log" ]; then
            cp -p ${logdir}/cluster_wd.log ${logdir}/cluster_wd.log.long_retention
        fi
    endscript
}
${logdir}/cluster_wd.log.long_retention {
    su ${osuser} ${osgroup}
    daily
    missingok
    copytruncate
    rotate 30
    compress
    notifempty
    nodateext
}

section_id: log-rotation


title: View and manage Redis slow log

title: Overview

text:
[Redis slow log]() is one of the best
tools for debugging and tracing your Redis database, especially if you
experience high latency and high CPU usage with Redis operations.
Because Redis is based on a single threaded architecture, Redis slow log
can be much more useful than slow log mechanisms of multi-threaded
database systems such as MySQL slow query log.

Unlike tools that introduce lock overhead, which complicates the debugging
process, Redis slow log is highly effective at showing the actual processing time of each command.

title: Redis Software slow log enhancements

text:
Redis Software includes enhancements to the standard Redis
slow log capabilities that allow you to analyze the execution time
complexity of each command. This enhancement can help you better analyze
Redis operations, allowing you to compare the differences between
execution times of the same command, observe spikes in CPU usage, and
more.

This is especially useful with complex commands such as
[ZUNIONSTORE](),
[ZINTERSTORE](), and
[ZRANGEBYSCORE]().

The enhanced Redis Software slow log adds the **Complexity info** field to the
output data.

View the complexity info data by its respective command in the table
below:

| Command | Value of interest | Complexity |
|------------|-----------------|-----------------|
| LINSERT | N - list len | O(N) |
| LREM | N - list len | O(N) |
| LTRIM | N - number of removed elements | O(N) |
| PUBLISH | N - number of channel subscribers
M - number of subscribed patterns | O(N+M) |
| PSUBSCRIBE | N - number of patterns client is subscribed to
argc - number of arguments passed to the command | O(argc\*N) |
| PUNSUBSCRIBE | N - number of patterns client is subscribed to
M - total number of subscribed patterns
argc - number of arguments passed to the command | O(argc\*(N+M)) |
| SDIFF | N - total number of elements in all sets | O(N) |
| SDIFFSTORE | N - total number of elements in all sets | O(N) |
| SINTER                | N - number of elements in smallest set
argc - number of arguments passed to the command | O(argc\*N) |
| SINTERSTORE           | N - number of elements in smallest set
argc - number of arguments passed to the command | O(argc\*N) |
| SMEMBERS              | N - number of elements in a set | O(N) |
| SORT                  | N - number of elements in the when no sorting list/set/zset
M - number of elements in result | O(N+M\*log(M))O(N) |
| SUNION                | N - number of elements in all sets | O(N) |
| SUNIONSTORE           | N - number of elements in all sets | O(N) |
| UNSUBSCRIBE           | N - total number of clients subscribed to all channels | O(N) |
| ZADD                  | N - number of elements in the zset | O(log(N)) |
| ZCOUNT                | N - number of elements in the zset
M - number of elements between min and max | O(log(N)+M) |
| ZINCRBY               | N - number of elements in the zset | O(log(N)) |
| ZINTERSTORE           | N – number of elements in the smallest zset
K – number of zsets
M – number of elements in the results set | O(N\*K)+O(M\*log(M)) |
| ZRANGE                | N – number of elements in the zset
M – number of results | O(log(N)+M) |
| ZRANGEBYSCORE         | N – number of elements in the zset
M – number of results | O(log(N)+M) |
| ZRANK                 | N – number of elements in the zset | O(log(N)) |
| ZREM                  | N – number of elements in the zset
argc – number of arguments passed to the command | O(argc\*log(N)) |
| ZREMRANGEBYRANK       | N – number of elements in the zset
argc – number of arguments passed to the command | O(log(N)+M) |
| ZREMRANGEBYSCORE      | N – number of elements in the zset
M – number of elements removed | O(log(N)+M) |
| ZREVRANGE             | N – number of elements in the zset
M – number of results | O(log(N)+M) |
| ZREVRANK              | N – number of elements in the zset | O(log(N)) |
| ZUNIONSTORE           | N – sum of element counts of all zsets
M – element count of result | O(N)+O(M\*log(M)) |

title: View slow log

text:
To view slow log entries for Redis Software databases, use one of the following methods:

- Cluster Manager UI:

    1. To access the slow log in the Cluster Manager UI, your [cluster management role]() must be Admin, Cluster Member, or DB Member.

    1. Select a database from the **Databases** list.

    1. On the database's **Configuration** screen, select the **Slowlog** tab.

- Command line:

    Use [`redis-cli`]() to run [`SLOWLOG GET`]():

    [code example]

title: Change slow log threshold

text:
The slow log includes all database commands that take longer than ten milliseconds (10,000 microseconds) by default. You can use [`redis-cli`]() to view or change this threshold.

To check the current threshold, run [`CONFIG GET`]():

[code example]

To change the threshold, run [`CONFIG SET`]():

[code example]

title: Change maximum entries

text:
The slow log retains the last 128 entries by default. You can use [`redis-cli`]() to view or change the maximum number of entries.

To check the current maximum, run [`CONFIG GET`]():

[code example]

To change the maximum, run [`CONFIG SET`]():

[code example]

code: redis-cli -h <endpoint> -p <port> SLOWLOG GET <count>

section_id: view-slow-log

code: redis-cli -h <endpoint> -p <port> CONFIG GET slowlog-log-slower-than

section_id: change-slow-log-threshold

code: redis-cli -h <endpoint> -p <port> CONFIG SET slowlog-log-slower-than <value_in_microseconds>

code: redis-cli -h <endpoint> -p <port> CONFIG GET slowlog-max-len

section_id: change-maximum-entries

code: redis-cli -h <endpoint> -p <port> CONFIG SET slowlog-max-len <value>


title: rsyslog logging

title: Log concepts

text:
Redis Software logs information from a variety of components in response to actions and events that occur within the cluster.

In some cases, a single action, such as removing a node from the cluster, may actually consist of several events. These actions may generate multiple log entries.

All log entries displayed in the Cluster Manager UI are also written to `syslog`.  You can configure `rsyslog` to monitor `syslog`. Enabled alerts are logged to `syslog` and appear with other log entries.

You can also [manage your logs]() with a remote logging server and log rotation.

title: Types of log entries

text:
Log entries are categorized into events and alerts. Both types of entries appear in the logs, but alert log entries also include a boolean `"state"` parameter that indicates whether the alert is enabled or disabled.

Log entries include information about the specific event that occurred. See the log entry tables for [alerts and events]() for more details.

title: Severity

text:
You can also configure `rsyslog` to add other information, such as the event severity.

Since `rsyslog` entries do not include severity by default, you can follow these steps to enable it:

1. Add the following line to `/etc/rsyslog.conf`:
    [code example]

2. Modify `$ActionFileDefaultTemplate` to use your new template `$ActionFileDefaultTemplateTraditionalFormatWithPRI`

3. Save these changes and restart `rsyslog` to apply them

You can see the log entries for alerts and events in the `/var/log/messages` file.

**Command components:**

- `%pritext%` adds the severity
- `%timegenerated%` adds the timestamp
- `%HOSTNAME%` adds the machine name
- `%syslogtag%` adds the Redis Software message. See the [log entry structure](#log-entry-structure) section for more details.
- `%msg:::droplastlf%n` removes duplicated log entries

title: Log entry structure

text:
The log entries have the following basic structure:

    event_log[<process id>]:{<list of key-value pairs in any order>}

- **event_log**: Plain static text is always shown at the beginning of the entry.
- **process id**: The ID of the logging process
- **list of key-value pairs in any order**: A list of key-value pairs that describe the specific event. They can appear in any order. Some key-value pairs are always shown, and some appear depending on the specific event.
    - **Key-value pairs that always appear:**
        - `"type"`: A unique code name for the logged event. For the list of codenames, see the [logged alerts and events]() tables.
        - `"object"`: Defines the object type and ID (if relevant) of the object this event relates to, such as cluster, node with ID, BDB with ID, etc. Has the format of <nobr>`<object type>[:<id>]`</nobr>.
        - `"time"`: Unix epoch time but can be ignored in this context.
    - **Key-value pairs that might appear depending on the specific entry:**
        - `"state"`: A boolean where `true` means the alert is enabled, and `false` means the alert is disabled. This is only relevant for alert log entries.
        - `"global_threshold"`: The value of a threshold for alerts related to cluster or node objects.
        - `"threshold"`: The value of a threshold for alerts related to a BDB object

title: Log entry samples

text: This section provides examples of log entries that include the [`rsyslog` configuration](#severity) to add the severity, timestamp, and machine name.

title: Ephemeral storage passed threshold

text:
#### "Alert on" log entry sample

[code example]

In this example, the storage utilization on node 1 reached the value of ~90%, which triggered the alert for "Ephemeral storage has reached 70% of its capacity."

**Log entry components:**

- `daemon.warning` - Severity of entry is `warning`
- `Jun 14 14:49:20` - The timestamp of the event
- `node1`: Machine name
- `event_log` - Static text that always appears
- `[3464]` - Process ID
- `"storage_util":90.061643120001` - Current ephemeral storage utilization
- `"global_threshold":"70"` - The user-configured threshold above which the alert is raised
- `"object":"node:1"` - The object related to this alert
- `"state":true` - Current state of the alert
- `"time":1434282560` - Can be ignored
- `"type":"ephemeral_storage"` - The code name of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

This log entry is an example of when the alert for the node with ID 1 "Ephemeral storage has reached 70% of its capacity" has been turned off as result of storage utilization reaching the value of ~60%.

**Log entry components**:

- `daemon.info` - Severity of entry is `info`
- `Jun 14 14:51:35` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[3464]` - Process ID
- `"storage_util":60.051723520008` - Current ephemeral storage utilization
- `"global_threshold":"70"` - The user configured threshold above which the alert is raised (70% in this case)
- `"object":"node:1"` - The object related to this alert
- `"state":false` - Current state of the alert
- `"time":1434283480` - Can be ignored
- `"type":"ephemeral_storage"` - The code name identifier of this specific event. See [logged alerts and events]() for more details.

title: Odd number of nodes with a minimum of three nodes alert

text:
#### "Alert on" log entry sample

[code example]

This log entry is an example of when the alert for "True high availability requires an odd number of nodes with a minimum of three nodes" has been turned on as result of the cluster having only one node.

**Log entry components:**

- `daemon.warning` - Severity of entry is warning
- `Jun 14 15:25:00` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[8310]` - Process ID
- `"object":"cluster"` - The object related to this alert
- `"state":true` - Current state of the alert
- `"time":1434284700` - Can be ignored
- `"node_count":1` - The number of nodes in the cluster
- `"type":"even_node_count"` - The code name identifier of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

This log entry is an example of when the alert for "True high availability requires an odd number of nodes with a minimum of three nodes" has been turned off as result of the cluster having 3 nodes.

**Log entry components:**

- `daemon.warning` - Severity of entry is warning
- `Jun 14 15:30:40` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[8310]` - Process ID
- `"object":"cluster"` - The object related to this alert
- `"state":false` - Current state of the alert
- `"time":1434285200` - Can be ignored
- `"node_count":3` - The number of nodes in the cluster
- `"type":"even_node_count"` - The code name of this specific event. See [logged alerts and events]() for more details.

text:
#### "Alert on" log entry sample

[code example]

This log entry is an example of when the alert for "Node has insufficient disk space for AOF rewrite" has been turned on as result of not having enough persistent storage disk space for AOF rewrite purposes. It is missing 604602126 bytes.

**Log entry components:**

- `daemon.err` - Severity of entry is error
- `Jun 15 13:51:23` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[34252]` - Process ID
- `"used":23457188` - The amount of disk space in bytes currently used for AOF files
- `"missing":604602126` - The amount of disk space in bytes that is currently missing for AOF rewrite purposes
- `"object":"node:1″` - The object related to this alert
- `"free":9867264` - The amount of disk space in bytes that is currently
    free
- `"needed":637926578` - The amount of total disk space in bytes that is needed for AOF rewrite purposes
- `"state":true` - Current state of the alert
- `"time":1434365483` - Can be ignored
- `"disk":705667072` - The total size in bytes of the persistent storage
- `"type":"insufficient_disk_aofrw"` - The code name of this specific event. See [logged alerts and events]() for more details.

#### "Alert off" log entry sample

[code example]

**Log entry components:**

- `daemon.info` - Severity of entry is info
- `Jun 15 13:51:11` - The timestamp of the event
- `node1` - Machine name
- `event_log` - Static text that always appears
- `[34252]` - Process ID
- `"used":0` - The amount of disk space in bytes currently used for AOF files
- `"missing":‐21614592` - The amount of disk space in bytes that is currently missing for AOF rewrite purposes. In this case, it is not missing because the number is negative.
- `"object":"node:1″` - The object related to this alert
- `"free":21614592` - The amount of disk space in bytes that is currently free
- `"needed":0` - The amount of total disk space in bytes that is needed for AOF rewrite purposes. In this case, no space is needed.
- `"state":false` - Current state of the alert
- `"time":1434365471` - Can be ignored
- `"disk":705667072` - The total size in bytes of the persistent storage
- `"type":"insufficient_disk_aofrw"` - The code name of this specific event. See [logged alerts and events]() for more details.

code: $template TraditionalFormatWithPRI,"%pri-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

section_id: severity

code:
daemon.warning: Jun 14 14:49:20 node1 event_log[3464]:
{
    "storage_util": 90.061643120001,
    "global_threshold": "70",
    "object": "node:1",
    "state": true,
    "time": 1434282560,
    "type": "ephemeral_storage"
}

section_id: ephemeral-storage-passed-threshold

code:
daemon.info: Jun 14 14:51:35 node1 event_log[3464]:
{
    "storage_util":60.051723520008,
    "global_threshold": "70",
    "object": "node:1",
    "state":false,
    "time": 1434283480,
    "type": "ephemeral_storage"
}

code:
daemon.warning: Jun 14 15:25:00 node1 event_log[8310]:
{
    "object":"cluster",
    "state": true,
    "time": 1434284700,
    "node_count": 1,
    "type":"even_node_count"
}

code:
daemon.warning: Jun 14 15:30:40 node1 event_log[8310]:
{
    "object":"cluster",
    "state": false,
    "time": 1434285200,
    "node_count": 3,
    "type":"even_node_count"
}


title: Maintenance mode for cluster nodes

summary: Prepare a cluster node for maintenance.

title: Overview

text: Use maintenance mode to prevent data loss during hardware patching or operating system maintenance on Redis Software servers. When maintenance mode is on, all shards move off of the node under maintenance and migrate to another available node.

title: Activate maintenance mode

text:
When you activate maintenance mode, Redis Software does the following:

1. Checks whether the cluster has enough active nodes to maintain quorum if one goes down. If taking a node offline would cause quorum loss, maintenance mode will not activate.

This check does not consider if other nodes are already in maintenance mode. Do not put the majority of nodes into maintenance mode and restart them simultaneously, as this can cause the cluster to lose quorum and lead to data loss.

1. If no maintenance mode snapshots already exist or if you use `overwrite_snapshot` when you activate maintenance mode, Redis Software creates a new node snapshot that records the node's shard and endpoint configuration.

1. Marks the node as a quorum node to prevent shards and endpoints from migrating to it.

    At this point, [`rladmin status`]() displays the node's shards field in yellow, which indicates that shards cannot migrate to the node.

1. Migrates shards and binds endpoints to other nodes, when space is available.

Maintenance mode does not demote a master node by default. The cluster elects a new master node when the original master node restarts.

Add the `demote_node` option to the `rladmin` command to [demote a master node](#demote-a-master-node) when you activate maintenance mode.

To activate maintenance mode for a node, run the following command:

[code example]

You can start server maintenance if:

- All shards and endpoints have moved to other nodes

- Enough nodes are still online to maintain quorum

title: Prevent replica shard migration

text:
If you do not have enough resources available to move all of the shards to other nodes, you can turn maintenance mode on without migrating the replica shards.

Before you prevent replica shard migration during maintenance mode, consider the following effects:

- Replica shards remain on the node during maintenance.

- If the maintenance node fails, the master shards do not have replica shards to maintain data redundancy and high availability.

- Replica shards that remain on the node can still be promoted during failover to preserve availability.

To activate maintenance mode without replica shard migration, run:

[code example]

title: Demote a master node

role: example

text:
If maintenance might affect connectivity to the master node, you can demote the master node when you  activate maintenance mode.  This lets the cluster elect a new master node.

To demote a master node when activating maintenance mode, run:

[code example]

title: Verify maintenance mode activation

text:
To verify maintenance mode for a node, use `rladmin status` and review the node's shards field.  If that value is displayed in yellow (shown earlier), then the node is in maintenance mode.

Avoid activating maintenance mode when it is already active.  Maintenance mode activations stack.  If you activate maintenance mode for a node that is already in maintenance mode, you will have to deactivate maintenance mode twice in order to restore full functionality.

title: Deactivate maintenance mode

text:
When you deactivate maintenance mode, Redis Software:

1. Loads a [specified node snapshot](#specify-a-snapshot) or defaults to the latest maintenance mode snapshot.

1. Unmarks the node as a quorum node to allow shards and endpoints to migrate to the node.

1. Restores the shards and endpoints that were in the node at the time of the snapshot.

1. Deletes the snapshot.

To deactivate maintenance mode after server maintenance, run:

[code example]

By default, a snapshot is required to deactivate maintenance mode.  If the snapshot cannot be restored, deactivation is cancelled and the node remains in maintenance mode.  In such events, it may be necessary to [reset node status](#reset_node_status).

title: Specify a snapshot

text:
When you turn off maintenance mode, you can restore the node configuration from a maintenance mode snapshot or any snapshots previously created by [`rladmin node <node_id> snapshot create`](). If you do not specify a snapshot, Redis Software uses the latest maintenance mode snapshot by default.

To get a list of available snapshots, run:

[code example]

To specify a snapshot when you turn maintenance mode off, run:

[code example]

If an error occurs when you turn on maintenance mode, the snapshot is not deleted.
When you rerun the command, use the snapshot from the initial attempt since it contains the original state of the node.

title: Skip shard restoration

text:
You can prevent the migrated shards and endpoints from returning to the original node after you turn off maintenance mode.

To turn maintenance mode off and skip shard restoration, run:

[code example]

title: Reset node status

text:
In extreme cases, you may need to reset a node's status.  Run the following commands to do so:

[code example]

Use these commands with caution.  For best results, contact Support before running these commands.

title: Cluster status example

text:
This example shows how the output of [`rladmin status`]() changes when you turn on maintenance mode for a node.

The cluster status before turning on maintenance mode:

[code example]

The cluster status after turning on maintenance mode:

[code example]

After turning on maintenance mode for node 2, Redis Software saves a snapshot of its configuration and then moves its shards and endpoints to node 3.

Now node 2 has `0/0` shards because shards cannot migrate to it while it is in maintenance mode.

title: Maintenance mode REST API

text: You can also turn maintenance mode on or off using [REST API requests]() to [<nobr>POST `/nodes/{node_uid}/actions/{action}`</nobr>]().

title: Activate maintenance mode (REST API)

text:
Use <nobr>`POST /nodes/{node_uid}/actions/maintenance_on`</nobr> to activate maintenance mode:

[code example]

You can set `evict_ha_replica` and `evict_active_active_replica` to `false` to [prevent replica shard migration](#prevent-replica-shard-migration).

The `maintenance_on` request returns a JSON response body:

[code example]

title: Deactivate maintenance mode (REST API)

text:
Use <nobr>`POST /nodes/{node_uid}/actions/maintenance_off`</nobr> deactivate maintenance mode:

[code example]

The `skip_shards_restore` boolean flag allows the `maintenance_off` action to [skip shard restoration](#skip-shard-restoration) when set to `true`.

The `maintenance_off` request returns a JSON response body:

[code example]

title: Track action status

text:
You can send a request to [<nobr>GET `/nodes/{node_uid}/actions/{action}`</nobr>]() to track the [status]() of the `maintenance_on` and `maintenance_off` actions.

This request returns the status of the `maintenance_on` action:

[code example]

The response body:

[code example]

code: rladmin node <node_id> maintenance_mode on overwrite_snapshot

section_id: activate-maintenance-mode

code: rladmin node <node_id> maintenance_mode on evict_ha_replica disabled evict_active_active_replica disabled

section_id: prevent-replica-shard-migration

code: rladmin node <node_id> maintenance_mode on demote_node

section_id: demote-a-master-node

section_id: deactivate-maintenance-mode

code: rladmin node <node_id> snapshot list

section_id: specify-a-snapshot

code: rladmin node <node_id> maintenance_mode off snapshot_name <snapshot_name>

code: rladmin node <node_id> maintenance_mode off skip_shards_restore

section_id: skip-shard-restoration

code:
$ rladmin tune node <node_id> max_listeners 100
$ rladmin tune node <node_id> quorum_only disabled

section_id: reset-node-status

code:
redislabs@rp1_node1:/opt$ rladmin status
CLUSTER NODES:
NODE:ID   ROLE     ADDRESS       EXTERNAL_ADDRESS     HOSTNAME    SHARDS
*node:1   master   172.17.0.2                         rp1_node1   2/100
node:2    slave    172.17.0.4                         rp3_node1   2/100
node:3    slave    172.17.0.3                         rp2_node1   0/100

section_id: cluster-status-example

code:
redislabs@rp1_node1:/opt$ rladmin node 2 maintenance_mode on
Performing maintenance_on action on node:2: 0%
created snapshot NodeSnapshot<name=maintenance_mode_2019-03-14_09-50-59,time=None,node_uid=2>

node:2 will not accept any more shards
Performing maintenance_on action on node:2: 100%
OK
redislabs@rp1_node1:/opt$ rladmin status
CLUSTER NODES:
NODE:ID   ROLE     ADDRESS       EXTERNAL_ADDRESS     HOSTNAME    SHARDS
*node:1   master   172.17.0.2                         rp1_node1   2/100
node:2    slave    172.17.0.4                         rp3_node1   0/0
node:3    slave    172.17.0.3                         rp2_node1   2/100

code:
POST https://<hostname>:9443/v1/nodes/<node_id>/actions/maintenance_on
{
    "overwrite_snapshot": true,
    "evict_ha_replica": true,
    "evict_active_active_replica": true
}

section_id: activate-maintenance-mode-rest-api

status: queued

task_id: <task-id-guid>

code:
POST https://<hostname>:9443/v1/nodes/<node_id>/actions/maintenance_off
{ "skip_shards_restore": false }

section_id: deactivate-maintenance-mode-rest-api

section_id: track-action-status

status: completed


title: Set up a new cluster

summary: How to set up a new cluster using the management UI.

title: Content

text:
A Redis Software cluster typically consists of several nodes.
For production deployments, we recommend an uneven number of nodes, with a minimum of three.

In a cluster that consists of only one node, some features and capabilities are not enabled,
such as database replication that provides high availability.

To set up a new cluster, you must first [install the Redis Software package]()
and then set up the cluster as described below.
After the cluster is created you can [add multiple nodes to the cluster]().

To create a cluster:

1. In a browser, go to `https://<name or IP address of the machine with Redis Software installed>:8443`.
    For example, if you installed Redis Software on a machine with IP address 10.0.1.34, go to <https://10.0.1.34:8443>.

- The management UI uses a [self-signed certificate for TLS encryption]().
- If the machine has both an internal IP address and an external IP address, use the external IP address to access the setup UI.

1. Select **Create new cluster**.

1. Enter an email and password for the administrator account, then select **Next** to proceed to cluster setup.

1. Enter your cluster license key if you have one. Otherwise, the cluster uses the trial license by default.

1. In the **Configuration** section:

    1. For **FQDN (Fully Qualified Domain Name)**, enter a unique name for the cluster.

        See the [instructions for DNS setup]()
        to make sure your cluster is reachable by name.

    1. Choose whether to [**Enable private & public endpoints support**]().

    1. Choose whether to [**Enable rack-zone awareness**]().

1. Click **Next**.

1. Configure storage and network settings:

    1. Enter a path for [*Ephemeral storage*](), or leave the default path.

    1. Enter a path for [*Persistent storage*](),
        or leave the default path.

    1. To enable [*Redis Flex or Auto Tiering*](),
        select **Enable flash storage** and enter the path to the flash storage.

    1. If the cluster is configured to support [rack-zone awareness](), set the **Rack-zone ID** for the new node.

    1. If your machine has multiple IP addresses, assign a single IPv4 type address for **Node-to-node communication (internal traffic)** and multiple IPv4/IPv6 type addresses for **External traffic**.

1. Select **Create cluster**.

1. Click **OK** to confirm that you are aware of the replacement of the HTTPS TLS certificate on the node,
    and proceed through the browser warning.

After a short wait, your cluster is created and you can sign in to the Cluster Manager UI.

You can now access any of the management capabilities, including:

- [Creating a new database]()
- [Joining a new node to a cluster]()


title: Disk sizing for heavy write scenarios

summary: Sizing considerations for persistent disk space for heavy throughput databases.

title: Overview

text:
In extreme write scenarios when append-only files (AOF) are enabled, the AOF rewrite process can require considerably more disk space for [database persistence]().

For disk size requirements for standard usage, see [Hardware requirements]().

title: Estimate required disk space

text:
To estimate the required persistent disk space for AOF rewrite purposes in extreme write scenarios, use the following formula:

**X (1 + 3Y + Y²)**

Where:
- **X** = size of each shard in GB
- **Y** = number of shards

title: Examples

text:
The following examples show how to calculate the persistent disk space required for heavy write scenarios for different database configurations, where:

- **Database size** is the memory limit configured for the database.

- **Number of shards** is the total number of shards (primary shards + replica shards).

title: Example 1

text:
- Database size = 10 GB
- Number of shards = 4

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 73 GB of required disk space.

title: Example 2

text:
- Database size = 10 GB
- Number of shards = 16

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 191 GB of required disk space.

title: Example 3

text:
- Database size = 40 GB
- Number of shards = 5

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Required disk space: 328 GB.

title: Example 4

text:
- Database size = 40 GB
- Number of shards = 15

1. Calculate the shard size:

    [code example]

1. Use the formula to calculate the required persistent disk space:

    [code example]

1. Round up to 724 GB of required disk space.

code:
Shard size = database size / number of shards
               = 10 GB / 4 shards
               = 2.5 GB per shard

code:
Shard size = database size / number of shards
               = 10 GB / 16 shards
               = 0.625 GB per shard

code:
Shard size = database size / number of shards
               = 40 GB / 5 shards
               = 8 GB per shard

code:
Shard size = database size / number of shards
               = 40 GB / 15 shards
               = 2.67 GB per shard


title: Optimize clusters

summary: Configuration changes and information you can use to optimize your performance and memory usage.

summary: Use the `memtier_benchmark` tool to perform a performance benchmark of Redis Software.

title: Benchmarking Redis Software

title: Disk sizing for heavy write scenarios

summary: Optimize your cluster environments for better performance.

title: Cluster environment optimization

summary: Turn off services to free memory and improve performance.

title: Turn off services to free system memory

summary: Use the Redis OSS Cluster API to improve performance and keep applications current with cluster topology changes.

title: OSS Cluster API

summary: Use the wait command to take full advantage of Redis Software's replication-based durability.

title: Use the WAIT command to improve data safety and durability

page_type: index


title: Benchmarking Redis Software

title: Overview

text:
Use the `memtier_benchmark` tool to perform a performance benchmark of Redis Software.

Prerequisites:

- Redis Software installed
- A cluster configured
- A database created

For help with the prerequisites, see the [Redis Software quickstart]().

It is recommended to run memtier_benchmark on a separate node that is
not part of the cluster being tested. If you run it on a node of the
cluster, be mindful that it affects the performance of both the
cluster and memtier_benchmark.

[code example]

This command instructs memtier_benchmark to connect to your Redis
Enterprise database and generates a load doing the following:

- A 50/50 Set to Get ratio
- Each object has random data in the value

title: Populate a database with testing data

text:
If you need to populate a database with some test data for a proof of
concept, or failover testing, etc. here is an example for you.

[code example]

This command instructs memtier_benchmark to connect to your Redis
Enterprise database and generates a load doing the following:

- Write objects only, no reads
- A 500 byte object
- Each object has random data in the value
- Each key has a random pattern, then a colon, followed by a
    random pattern.

Run this command until it fills up your database to where you want it
for testing. The easiest way to check is on the database metrics page.

Another use for memtier_benchmark is to populate a database with data
for failure testing.

code: /opt/redislabs/bin/memtier_benchmark -s $DB_HOST -p $DB_PORT -a $DB_PASSWORD -t 4 -R --ratio=1:1

section_id: overview

code: /opt/redislabs/bin/memtier_benchmark -s $DB_HOST -p $DB_PORT -a $DB_PASSWORD -R -n allkeys -d 500 --key-pattern=P:P --ratio=1:0

section_id: populate-a-database-with-testing-data


title: Cluster environment optimization

title: Overview

text:
Redis Software uses various algorithms to optimize
performance. As part of this process, Redis Software examines usage
and load to adjust its runtime configuration. Depending
on your specific usage and load, Redis Software might take some
time to adjust for optimal performance.

To ensure optimal performance, you must run your workload several times
and for a long duration until performance stabilizes.

title: Failure detection sensitivity policies

text:
You can optimize your cluster's thresholds and timeouts for different environments using the `failure_detection_sensitivity` cluster policy:

- `high` (previously known as `local-network watchdog_profile`) – high failure detection sensitivity, lower thresholds, and faster failure detection and failover

- `low` (previously known as `cloud watchdog_profile`) – low failure detection sensitivity and higher tolerance for latency variance (also called network jitter)

Depending on which policy you choose, Redis Software uses different
thresholds to make operation-related decisions.

The default policy is `high` failure detection sensitivity for `local-network` environments. If you are
running Redis Software in a cloud environment, you should change the
configuration.

title: Change failure detection sensitivity

text:
To change the cluster's `failure_detection_sensitivity`, run one of the following [`rladmin`]() commands.

- For Redis Software version 6.4.2-69 and later, run:

    [code example]

- For Redis Software version 6.4.2-61 and earlier, run:

    [code example]

If Redis Software still
does not meet your performance expectations after following these instructions, [contact support](https://redis.com/company/support/) for help optimizing your environment.

code: rladmin tune cluster failure_detection_sensitivity [ low | high ]

section_id: change-failure-detection-sensitivity

code: rladmin tune cluster watchdog_profile [ cloud | local-network ]


title: OSS Cluster API

title: Content

text:
You can use the Redis OSS Cluster API along with other Redis Enterprise Software high availability
to get high performance with low latency
and let applications stay current with cluster topology changes, including add node, remove node, and node failover.

For more about working with the OSS Cluster API in Redis Software, see [Enable OSS Cluster API]().

To learn how to enable OSS Cluster API in Redis Cloud, see [Clustering Redis databases]().

To enable OSS Cluster API in Kubernetes, see [Enable cluster-aware clients (OSS Cluster API)]().


title: Turn off services to free system memory

title: Content

text:
The Redis Software cluster nodes host a range of services that support the cluster processes.
In most deployments, either all of these services are required,
or there are enough memory resources on the nodes for the database requirements.

In a deployment with limited memory resources, certain services can be disabled from API endpoint to free system memory or using the `rladmin` command.
Before you turn off a service, make sure that your deployment does not depend on that service.
After you turn off a service, you can re-enable in the same way.

The services that you can turn off are:

- RS Admin Console - `cm_server`
- Logs in CSV format - `stats_archiver`
- [LDAP authentication]() - `saslauthd`
- [Discovery service]()- `mdns_server`, `pdns_server`
- [Active-Active databases]() - `crdb_coordinator`, `crdb_worker`
- Alert Manager - `alert_mgr` (For best results, disable only if you have an alternate alert system.)

To turn off a service with the `rladmin cluster config` command, use the `services` parameter and the name of the service, followed by `disabled`.
[code example]

To turn off a service with the API, use the [`PUT /v1/services_configuration`]() endpoint
with the name of the service and the operating mode (enabled/disabled) in JSON format.

For example:
- To turn off the Redis Software Cluster Manager UI, use this PUT request:

    [code example]

- To turn off the CRDB services and enable the `stats_archiver` for cluster component statistics, use this PUT request:

    [code example]

code:
rladmin cluster config
        [ services <service_name> <enabled | disabled> ]

section_id: content

code:
PUT https://[host][:9443]/v1/cluster/services_configuration
    '{
        "cm_server":{
            "operating_mode":"disabled"
        }
    }'

code:
PUT https://[host][:9443]/v1/cluster/services_configuration
    '{
        "crdb_coordinator":{
            "operating_mode":"disabled"
        },
        "crdb_worker":{
            "operating_mode":"disabled"
        },
        "stats_archiver":{
            "operating_mode":"enabled"
        }
    }'


title: Overview

text:
Redis Software comes with the ability to replicate data
to another replica for high availability and persist in-memory data on
disk permanently for durability. With the [`WAIT`]() command, you can
control the consistency and durability guarantees for the replicated and
persisted database.

title: Non-blocking Redis write operation

text:
Any updates that are issued to the database are typically performed with the following flow:

1. Application issues a write.
1. Proxy communicates with the correct primary shard in the system that contains the given key.
1. The acknowledgment is sent to proxy after the write operation completes.
1. The proxy sends the acknowledgment back to the application.

Independently, the write is communicated from the primary shard to the replica, and
replication acknowledges the write back to the primary shard. These are steps 5 and 6.

Independently, the write to a replica is also persisted to disk and
acknowledged within the replica. These are steps 7 and 8.

title: Blocking write operation on replication

text:
With the [`WAIT`]() or [`WAITAOF`]() commands, applications can ask to wait for
acknowledgments only after replication or persistence is confirmed on
the replica. The flow of a write operation with `WAIT` or `WAITAOF` is:

1. The application issues a write.
1. The proxy communicates with the correct primary shard in the system that contains the given key.
1. Replication communicates the update to the replica shard.
1. If using `WAITAOF` and the AOF every write setting, the replica persists the update to disk before sending the acknowledgment.
1. The acknowledgment is sent back from the replica all the way to the proxy with steps 5 to 8.

The application only gets the acknowledgment from the write after durability is achieved with replication to the replica for `WAIT` or `WAITAOF` and to the persistent storage for `WAITAOF` only.

The `WAIT` command always returns the number of replicas that acknowledged the write commands sent by the current client before the `WAIT` command, both in the case where the specified number of replicas are reached, or when the timeout is reached. In Redis Software, the number of replicas is always 1 for databases with high availability enabled.

See the [`WAITAOF`]() command for details for enhanced data safety and durability capabilities introduced with Redis 7.2.


title: Remove a node

summary: Remove a node from your Redis Software cluster.

title: Overview

text:
You might want to remove a node from a Redis Software cluster for one of the following reasons:

- To [permanently remove a node](#permanently-remove-a-node) if you no longer need the extra capacity.
- To [replace a faulty node](#replace-a-faulty-node) with a healthy node.
- To [replace a healthy node](#replace-a-healthy-node) with a different node.

You can configure [email alerts from the cluster]() to notify you of cluster changes, including when a node is removed.

Read through these explanations thoroughly before taking
any action.

title: Permanently remove a node

text:
Permanently removing a node means you are decreasing cluster capacity.
Before trying to remove a node, make sure that the cluster has enough
capacity for all resources without that node, otherwise you cannot remove the node.

If there is not enough capacity in the cluster to facilitate removing
the node, you can either delete databases or add another node instead of
the one you would like to remove.

During the removal process, the cluster migrates all resources from the
node being removed to other nodes in the cluster. In order to ensure
database connectivity, and database high availability (when replication
is enabled), the cluster first creates replacement shards or endpoints
on one of the other nodes in the cluster, initiates failover as needed,
and only then removes the node.

If a cluster has only two nodes (which is not recommended for production
deployments) and some databases have replication enabled, you cannot remove a node.

title: Replace a faulty node

text:
If the cluster has a faulty node that you would like to replace, you
only need to add a new node to the cluster. The cluster recognizes the
existence of a faulty node and automatically replaces the faulty node
with the new node.

For guidelines, refer to [Replacing a faulty
node]().

title: Replace a healthy node

text:
If you would like to replace a healthy node with a different node, you
must first add the new node to the cluster, migrate all the resources
from the node you would like to remove, and only then remove the node.

For further guidance, refer to [adding a new node to a
cluster]().

You can migrate resources by using the `rladmin` command-line interface
(CLI). For guidelines, refer to [`rladmin` command-line interface
(CLI)]().

The [DNS records]() must be updated each time a node is added or replaced.

title: Remove a node

text:
To remove a node using the Cluster Manager UI:

1. On the **Nodes** screen, click  for the node you want to remove.

1. Select **Remove node** from the list.

1. If there are shards on the node, select **Confirm node removal**.

    If the node has no shards, the **Confirm node removal** checkbox does not appear.

1. Click **Remove**.

    Redis Software examines the node and the cluster, then takes the actions required to remove the node, such as migrating shards to other nodes. After the process finishes, the node is no longer shown in the UI.

At any point, you can click the **Cancel removal** button to stop the process. When cancelled, the current internal action is completed, and then the process stops.

To remove a node using the REST API, use [`POST /v1/nodes/<node_id>/actions/remove`]().

By default, the remove node action completes after all resources migrate off the removed node. Node removal does not wait for migrated shards' persistence files to be created on the new nodes.

To change node removal to wait for the creation of new persistence files for all migrated shards, set `wait_for_persistence` to `true` in the request body or [update the cluster policy]() `persistent_node_removal` to `true` to change the cluster's default behavior.

For example:

[code example]

If you need to add a removed node back to the cluster,
you must [uninstall]()
and [reinstall]() the software on that node.

code:
POST https://<hostname>:9443/v1/nodes/<node_id>/actions/remove
{
    "wait_for_persistence": true
}

section_id: remove-a-node


title: Replace a cluster node

summary: Replace a node in your cluster that is down.

title: Content

text:
A failed node will appear as `Down` () in the **Nodes** list.

To replace a failed node:

1. Prepare a new node identical to the old one.

1.  Install and
    configure Redis Software on the node. See [Install and setup]() for more information.

If you are using [Redis Flex or Auto Tiering](), make sure the required flash storage is set up on this new node.

1. [Add the node]() to the cluster. Make sure the new node has as much available memory as the faulty
    node.

    If the new node does not have enough memory, you will be prompted to add a node with enough memory.

1. A message will appear informing you that the cluster has a faulty node
    and that the new node will replace the faulty node.

- If there is a faulty node in the cluster to which you are adding a node, Redis Software will use the new node to replace the faulty one.
- Any existing [DNS records]() must be updated
each time a node is added or replaced.


title: Enable causal consistency

summary: Enable causal consistency in an Active-Active database.

title: Overview

text:
When you enable causal consistency in Active-Active databases,
the order of operations on a specific key are maintained across all Active-Active database instances.

For example, if operations A and B were applied on the same key and the effect of A was observed by the instance that initiated B before B was applied to the key.
All instances of an Active-Active database would then observe the effect of A before observing the effect of B.
This way, any causal relationship between operations on the same key is also observed and maintained by every replica.

title: Enable causal consistency

text:
When you create an Active-Active database, you can enable causal consistency in the Cluster Manager UI:

1. In the **Participating clusters** section of the **Create Active-Active database** screen, locate **Causal Consistency**:

1. Click **Change** to open the **Causal Consistency** dialog.

1. Select **Enabled**:

1. Click **Change** to confirm your selection.

After database creation, you can only turn causal consistency on or off using the REST API or `crdb-cli`.
The updated setting only affects commands and operations received after the change.

title: Causal consistency side effects

text:
When the causal consistency option is enabled, each instance maintains the order of operations it received from another instance
and relays that information to all other N-2 instances,
where N represents the number of instances used by the Active-Active database.

As a result, network traffic is increased by a factor of (N-2).
The memory consumed by each instance and overall performance are also impacted when causal consistency is activated.


title: Connect to your Active-Active databases

summary: How to connect to an Active-Active database using redis-cli or a sample Python application.

title: Overview

text:
With the Redis database created, you are ready to connect to your
database to store data. You can use one of the following ways to test
connectivity to your database:

- Connect with redis-cli, the built-in command-line tool
- Connect with a _Hello World_ application written in Python

Remember we have two member Active-Active databases that are available for connections and
concurrent reads and writes. The member Active-Active databases are using bi-directional
replication to for the global Active-Active database.

title: Connecting using redis-cli {#connecting-using-rediscli}

text:
redis-cli is a simple command-line tool to interact with redis database.

1. To use redis-cli on port 12000 from the node 1 terminal, run:

    [code example]

1. Store and retrieve a key in the database to test the connection with these
    commands:

    - `set key1 123`
    - `get key1`

    The output of the command looks like this:

    [code example]

1. Enter the terminal of node 1 in cluster 2, run the redis-cli, and
   retrieve key1.

    The output of the commands looks like this:

    [code example]

title: Connecting using _Hello World_ application in Python

text:
A simple python application running on the host machine can also connect
to the database.

Before you continue, you must have python and
[redis-py](https://github.com/andymccurdy/redis-py#installation)
(python library for connecting to Redis) configured on the host machine
running the container.

1. In the command-line terminal, create a new file called "redis_test.py"

    [code example]

1. Paste this code into the "redis_test.py" file.

    This application stores a value in key1 in cluster 1, gets that value from
    key1 in cluster 1, and gets the value from key1 in cluster 2.

    [code example]

1. To run the "redis_test.py" application, run:

    [code example]

    If the connection is successful, the output of the application looks like:

    [code example]

code: redis-cli -p 12000

code: vi redis_test.py

code:
import redis
    rp1 = redis.StrictRedis(host='localhost', port=12000, db=0)
    rp2 = redis.StrictRedis(host='localhost', port=12002, db=0)
    print ("set key1 123 in cluster 1")
    print (rp1.set('key1', '123'))
    print ("get key1 cluster 1")
    print (rp1.get('key1'))
    print ("get key1 from cluster 2")
    print (rp2.get('key1'))

code: python redis_test.py

code:
set key1 123 in cluster 1
    True
    get key1 cluster 1
    "123"
    get key1 from cluster 2
    "123"


title: Create an Active-Active geo-replicated database

summary: How to create an Active-Active database and things to consider when setting it up.

title: Overview

text:
[Active-Active geo-replicated databases]() (formerly known as CRDBs) give applications write access
to replicas of the dataset in different geographical locations.

The participating Redis Software clusters that host the instances can be distributed in different geographic locations.
Every instance of an Active-Active database can receive write operations, and all operations are [synchronized]() to all instances without conflict.

title: Steps to create an Active-Active database

text:
1. **Create a service account** - On each participating cluster, create a dedicated user account with the Admin role.
1. **Confirm connectivity** - Confirm network connectivity between the participating clusters.
1. **Create Active-Active database** - Connect to one of your clusters and create a new Active-Active database.
1. **Add participating clusters** - Add the participating clusters to the Active-Active database with the user credentials for the service account.
1. **Verify creation** - Log in to each of the participating clusters and verify your Active-Active database was created on them.
1. **Confirm Active-Active database synchronization** - Test writing to one cluster and reading from a different cluster.

title: Prerequisites

text:
- Two or more machines with the same version of Redis Software installed
- Network connectivity and cluster FQDN name resolution between all participating clusters
- [Network time service]() listener (ntpd) configured and running on each node in all clusters

title: Create an Active-Active database

text:
1. Create service accounts on each participating cluster:

    1. In a browser, open the Cluster Manager UI for the participating cluster.

        The default address is: `https://<RS_address>:8443`

    1. Go to the **Access Control > Users** tab:

    1. Click **+ Add user**.

    1. Enter the username, email, and password for the user.

    1. Select the **Admin** role.

    1. Click **Save**.

1. To verify network connectivity between participating clusters,
    run the following `telnet` command from each participating cluster to all other participating clusters:

    [code example]

1. In a browser, open the Cluster Manager UI of the cluster where you want to create the Active-Active database.

    The default address is: `https://<RS_address>:8443`

1. Open the **Create database** menu with one of the following methods:

    - Click the **+** button next to **Databases** in the navigation menu:

    - Go to the **Databases** screen and select **Create database**:

1. Select **Active-Active database**.

1. Enter the cluster's local admin credentials, then click **Save**:

1. Add participating clusters that will host instances of the Active-Active database:

    1. In the **Participating clusters** section, go to **Other participating clusters** and click **+ Add cluster**.

    1. In the **Add cluster** configuration panel, enter the new cluster's URL, port number, and the admin username and password for the new participating cluster:

If an Active-Active database [runs on flash memory](), you cannot add participating clusters that run on RAM only.

    1. Click **Join cluster** to add the cluster to the list of participating clusters.

1. Enter a **Database name**.

1. If your cluster supports [Redis Flex or Auto Tiering](), in **Runs on** you can select **Flash** so that your database uses Flash memory. We recommend that you use AOF every 1 sec for the best performance during the initial Active-Active database sync of a new replica.

1. To configure additional database settings, expand each relevant section to make changes.

    See [Configuration settings](#configuration-settings) for more information about each setting.

1. Click **Create**.

title: Configuration settings

text:
- **Database version** - The Redis version used by your database.

- **Database name** - The database name requirements are:

    - Maximum of 63 characters

    - Only letters, numbers, or hyphens (-) are valid characters

    - Must start and end with a letter or digit

    - Case-sensitive

- **Port** - You can define the port number that clients use to connect to the database. Otherwise, a port is randomly selected.

You cannot change the [port number]()
after the database is created.

- **Memory limit** - [Database memory limits]() include all database replicas and shards, including replica shards in database replication and database shards in database clustering.

    If the total size of the database in the cluster reaches the memory limit, the data eviction policy for the database is enforced.

If you create a database with Auto Tiering enabled, you also need to set the RAM-to-Flash ratio
for this database. Minimum RAM is 10%. Maximum RAM is 50%.

- **Memory eviction** - The default [eviction policy]() for Active-Active databases is `noeviction`. Redis Software versions 6.0.20 and later support all eviction policies for Active-Active databases, unless [Redis Flex or Auto Tiering]() is enabled.

- [**Capabilities**]() (previously **Modules**) - When you create a new in-memory database, you can enable additional capabilities in the database. You cannot enable them after database creation.

    Active-Active databases created with or upgraded to Redis version 8 or later automatically enable [search and query]() and [JSON](), which allows you to index, query, and perform full-text searches of nested JSON documents.

    For Auto Tiering databases, you can enable capabilities that support Auto Tiering. See [Redis Software and Redis Stack feature compatibility
]() for compatibility details.

    To add capabilities to the database:

    1. In the **Capabilities** section, select one or more capabilities.

    1. To customize capabilities, select **Parameters** and enter the optional custom configuration.

    1. Select **Done**.

title: TLS

text:
If you enable TLS when you create the Active-Active database, the nodes use the TLS mode **Require TLS for CRDB communication only** to require TLS authentication and encryption for communications between participating clusters.

After you create the Active-Active database, you can set the TLS mode to **Require TLS for all communications** so client communication from applications are also authenticated and encryption.

title: High availability

text:
- [**Replication**]() - We recommend that all Active-Active database use replication for best intercluster synchronization performance.

    When replication is enabled, every Active-Active database master shard is replicated to a corresponding replica shard. The replica shards are then used to synchronize data between the instances, and the master shards are dedicated to handling client requests.

- [**Replica high availability**]() - We also recommend that you enable replica high availability to ensure replica shards are highly-available for this synchronization.

title: Clustering

text:
- In the [**Clustering**]() section, you can either:

    - **Enable sharding** and select the number of shards you want to have in the database. When database clustering is enabled, databases have limitations for [multi-key operations]().

        You can increase the number of shards in the database at any time.

    - Clear the **Enable sharding** option to use only one shard, which allows you to use [multi-key operations]() without the limitations.

You cannot enable or turn off database clustering after the Active-Active database is created.

- [**OSS Cluster API**]() - The OSS Cluster API configuration allows access to multiple endpoints for increased throughput. The OSS Cluster API setting applies to all instances of the Active-Active database across participating clusters.

    This configuration requires clients to connect to the primary node to retrieve the cluster topology before they can connect directly to proxies on each node.

    When you enable the OSS Cluster API, shard placement changes to _Sparse_, and the database proxy policy changes to _All primary shards_ automatically.

title: Durability

text:
To protect against loss of data stored in RAM, you can enable [**Persistence**]() to store a copy of the data on disk.

Active-Active databases support append-only file (AOF) persistence only. Snapshot persistence is not supported for Active-Active databases.

title: Access control

text:
- **Unauthenticated access** - You can access the database as the default user without providing credentials.

- **Password-only authentication** - When you configure a password for your database's default user, all connections to the database must authenticate with the [AUTH command](.

    If you also configure an access control list, connections can specify other users for authentication, and requests are allowed according to the Redis ACLs specified for that user.

    Creating a database without ACLs enables a *default* user with full access to the database. You can secure default user access by requiring a password.

- **Access Control List** - You can specify the [user roles]() that have access to the database and the [Redis ACLs]() that apply to those connections.

    You can only configure access control after the Active-Active database is created. In each participating cluster, add ACLs after database creation.

    To define an access control list for a database:

    1. In **Security > Access Control > Access Control List**, select **+ Add ACL**.

    1. Select a [role]() to grant database access.

    1. Associate a [Redis ACL]() with the role and database.

    1. Select the check mark to add the ACL.

title: Causal consistency

text:
[**Causal consistency**]() in an Active-Active database guarantees that the order of operations on a specific key is maintained across all instances of an Active-Active database.

To enable causal consistency for an existing Active-Active database, use the REST API.

<!-- Also in getting-started-crdbs.md -->

title: Test Active-Active database connections

text: With the Redis database created, you are ready to connect to your database. See [Connect to Active-Active databases]() for tutorials and examples of multiple connection methods.

code: telnet <target FQDN> 9443

section_id: create-an-active-active-database


title: Delete Active-Active databases

summary: Considerations while deleting Active-Active databases.

title: Content

text:
When you delete an Active-Active database (formerly known as CRDB),
all instances of the Active-Active database are deleted from all participating clusters.

This action is immediate, non-reversible, and has no rollback.

Because Active-Active databases are made up of instances on multiple participating clusters,
to restore a deleted Active-Active database you must create the database again with all of its instances
and then restore the data to the database from backup.

We recommended that you:

- Back up your data and test the restore on another database before you delete an Active-Active database.
- Consider [flushing the data]() from the database
    so that you can keep the Active-Active database configuration and restore the data to it if necessary.


title: Application failover with Active-Active databases

summary: How to failover your application to connect to a remote replica.

title: Overview

text:
Active-Active Redis deployments don't have a built-in failover or failback mechanism for application connections.
An application deployed with an Active-Active database connects to a replica of the database that is geographically nearby.
If that replica is not available, the application can failover to a remote replica, and failback again if necessary.
In this article we explain how this process works.

For other disaster recovery strategies including network-based, proxy-based, and client library approaches, see [Active-Active disaster recovery strategies]().

Active-Active connection failover can improve data availability, but can negatively impact data consistency.
Active-Active replication, like Redis replication, is asynchronous.
An application that fails over to another replica can miss write operations.
If the failed replica saved the write operations in persistent storage,
then the write operations are processed when the failed replica recovers.

title: Detecting Failure

text:
Your application can detect two types of failure:

1. **Local failures** - The local replica is down or otherwise unavailable
1. **Replication failures** - The local replica is available but fails to replicate to or from remote replicas

You can also use [database availability API requests]() to determine if a database replica is available to handle read and write operations. The lag-aware database availability requests considers CRDT replication lag as a health check criterion to prevent reading stale data during failback scenarios.

title: Local Failures

text: Local failure is detected when the application is unable to connect to the database endpoint for any reason. Reasons for a local failure can include: multiple node failures, configuration errors, connection refused, connection timed out, unexpected protocol level errors.

title: Replication Failures

text:
Replication failures are more difficult to detect reliably without causing false positives. Replication failures can include: network split, replication configuration issues, remote replica failures.

The most reliable method for health-checking replication is by using the Redis publish/subscribe (pub/sub) mechanism.

Note that this document does not suggest that Redis pub/sub is reliable in the common sense. Messages can get lost in certain conditions, but that is acceptable in this case because typically the application determines that replication is down only after not being able to deliver a number of messages over a period of time.

When you use the pub/sub data type to detect failures, the application:

1. Connects to all replicas and subscribes to a dedicated channel for each replica.
1. Connects to all replicas and periodically publishes a uniquely identifiable message.
1. Monitors received messages and ensures that it is able to receive its own messages within a predetermined window of time.

You can also use known dataset changes to monitor the reliability of the replication stream,
but pub/sub is preferred method because:

1. It does not involve dataset changes.
1. It does not make any assumptions about the dataset.
1. Pub/sub messages are delivered as replicated effects and are a more reliable indicator of a live replication link. In certain cases, dataset keys may appear to be modified even if the replication link fails. This happens because keys may receive updates through full-state replication (re-sync) or through online replication of effects.

title: Impact of sharding on failure detection

text:
If your sharding configuration is symmetric, make sure to use at least one key (PUB/SUB channels or real dataset key) per shard. Shards are replicated individually and are vulnerable to failure. Symmetric sharding configurations have the same number of shards and hash slots for all replicas.
We do not recommend an asymmetric sharding configuration, which requires at least one key per hash slot that intersects with a pair of shards.

To make sure that there is at least one key per shard, the application should:

1. Use the Cluster API to retrieve the database sharding configuration.
1. Compute a number of key names, such that there is one key per shard.
1. Use those key names as channel names for the pub/sub mechanism.

title: Failing over

text:
When the application needs to failover to another replica, it should simply re-establish its connections with the endpoint on the remote replica. Because Active/Active and Redis replication are asynchronous, the remote endpoint may not have all of the locally performed and acknowledged writes.

It's best if your application doesn't read its own recent writes. Those writes can be either:

1. Lost forever, if the local replica has an event such as a double failure or loss of persistent files.
1. Temporarily unavailable, but will be available at a later time if the local replica's failure is temporary.

<!---
Sample code that maps a hash slot to a key name can be found in this Python script.
 --->

title: Failback decision

text:
Your application can use the same checks described above to continue monitoring the state of the failed replica after failover.

To monitor the state of a replica during the failback process, you must make sure the replica is available, re-synced with the remote replicas, and not in stale mode. The PUB/SUB mechanism is an effective way to monitor this.

Dataset-based mechanisms are potentially less reliable for several reasons:
1. In order to determine that a local replica is not stale, it is not enough to simply read keys from it. You must also attempt to write to it.
1. As stated above, remote writes for some keys appear in the local replica before the replication link is back up and while the replica is still in stale mode.
1. A replica that was never written to never becomes stale, so on startup it is immediately ready but serves stale data for a longer period of time.

title: Replica Configuration Changes

text:
All failover and failback operations should be done strictly on the application side, and should not involve changes to the Active-Active configuration.
The only valid case for re-configuring the Active-Active deployment and removing a replica is when memory consumption becomes too high because garbage collection cannot be performed.
Once a replica is removed, it can only be re-joined as a new replica and it loses any writes that were not converged.


title: Hashes in an Active-Active databases

summary: Information about using hashes with an Active-Active database.

title: Content

text:
Hashes are great for structured data that contain a map of fields and
values. They are used for managing distributed user or app session
state, user preferences, form data and so on. Hash fields contain string
type and string types operate just like the standard Redis string types
when it comes to CRDTs. Fields in hashes can be initialized as a string
using HSET or HMSET or can be used to initialize counter types that are
numeric integers using HINCRBY or floats using HINCRBYFLOAT.

Hashes in Active-Active databases behave the same and maintain additional metadata to
achieve an "OR-Set" behavior to handle concurrent conflicting writes.
With the OR-Set behavior, writes to add new fields across multiple Active-Active database
instances are typically unioned except in cases of conflicts.
Conflicting instance writes can happen when an Active-Active database instance deletes a
field while the other adds the same field. In this case and observed
remove rule is followed. That is, remove can only remove fields it has
already seen and in all other cases element add/update wins.

Field values behave just like CRDT strings. String values can be types
string, counter integer based on the command used for initialization of
the field value. See "String Data Type in Active-Active databases" and "String Data Type
with Counter Value in Active-Active databases" for more details.

Here is an example of an "add wins" case:

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | HSET key1 field1 "a" |  |
|  t2 |  | HSET key1 field2 "b" |
|  t4 | - Sync - | - Sync - |
|  t5 | HGETALL key1
1) "field2"
2) "b"
3) "field1"
4) "a" | HGETALL key1
1) "field2"
2) "b"
3) "field1"
4) "a" |


title: HyperLogLog in Active-Active databases

summary: Information about using hyperloglog with an Active-Active database.

title: Overview

text:
**HyperLogLog** is an algorithm that addresses the [count-distinct problem](https://en.wikipedia.org/wiki/Count-distinct_problem).
To do this it approximates the numbers of items in a [set](https://en.wikipedia.org/wiki/Multiset).
Determining the _exact_ cardinality of a set requires memory according to the cardinality of the set.
Because it estimates the cardinality by probability, the HyperLogLog algorithm can run with more reasonable memory requirements.

title: HyperLogLog in Redis

text:
Redis Open source implements [HyperLogLog](https://redislabs.com/redis-best-practices/counting/hyperloglog/) (HLL) as a native data structure.
It supports adding elements ([PFADD]() to an HLL, counting elements ([PFCOUNT]() of HLLs, and merging of ([PFMERGE]() HLLs.

Here is an example of a simple write case:

| Time | Replica 1         | Replica 2         |
| ---- | ----------------- | ----------------- |
| t1   | PFADD hll x       |                   |
| t2   | --- sync ---      |                   |
| t3   |                   | PFADD hll y       |
| t4   | --- sync ---      |                   |
| t5   | PFCOUNT hll --> 2 | PFCOUNT hll --> 2 |

Here is an example of a concurrent add case:

| Time | Replica 1         | Replica 2         |
| ---- | ----------------- | ----------------- |
| t1   | PFADD hll x       | PFADD hll y       |
| t2   | PFCOUNT hll --> 1 | PFCOUNT hll --> 1 |
| t3   | --- sync ---      |                   |
| t4   | PFCOUNT hll --> 2 | PFCOUNT hll --> 2 |

title: The DEL-wins approach

text:
Other collections in the Redis-CRDT implementation use the observed remove method to resolve conflicts.
The CRDT-HLL uses the DEL-wins method.
If a DEL request is received at the same time as any other request (ADD/MERGE/EXPIRE) on the HLL-key
the replicas consistently converge to delete key.
In the observed remove method used by other collections (sets, lists, sorted-sets and hashes),
only the replica that received the DEL request removes the elements, but elements added concurrently in other replicas exist in the consistently converged collection.
We chose to use the DEL-wins method for the CRDT-HLL to maintain the original time and space complexity of the HLL in Redis Open source.

Here is an example of a DEL-wins case:

| HLL  |                 |                 | \|  | Set  |                     |                     |
| ---- | --------------- | --------------- | --- | ---- | ------------------- | ------------------- |
|      |                 |                 | \|  |      |                     |                     |
| Time | Replica 1       | Replica 2       | \|  | Time | Replica 1           | Replica 2           |
|      |                 |                 | \|  |      |                     |                     |
| t1   | PFADD h e1      |                 | \|  | t1   | SADD s e1           |                     |
| t2   | --- sync ---    |                 | \|  | t2   | --- sync ---        |                     |
| t3   | PFCOUNT h --> 1 | PFCOUNT h --> 1 | \|  | t3   | SCARD s --> 1       | SCARD s --> 1       |
| t4   | PFADD h e2      | Del h           | \|  | t4   | SADD s e2           | Del S               |
| t5   | PFCOUNT h --> 2 | PFCOUNT h --> 0 | \|  | t5   | SCARD s --> 2       | SCARD s --> 0       |
| t6   | --- sync ---    |                 | \|  | t6   | --- sync ---        |                     |
| t7   | PFCOUNT h --> 0 | PFCOUNT h --> 0 | \|  | t7   | SCARD s --> 1       | SCARD s --> 1       |
| t8   | Exists h --> 0  | Exists h --> 0  | \|  | t8   | Exists s --> 1      | Exists s --> 1      |
|      |                 |                 | \|  | t9   | SMEMBERS s --> {e2} | SMEMBERS s --> {e2} |

title: HLL in Active-Active databases versus HLL in Redis Open source

text:
In Active-Active databases, we implemented HLL within the CRDT on the basis of the Redis implementation with a few exceptions:

- Redis keeps the HLL data structure as an encoded string object
    such that you can potentially run any string request can on a key that contains an HLL. In CRDT, only get and set are supported for HLL.
- In CRDT, if you do SET on a key that contains a value encoded as an HLL, then the value will remain an HLL. If the value is not encoded as HLL, then it will be a register.


title: Data types for Active-Active databases

summary: Introduction to differences in data types between standalone and Active-Active Redis databases.

title: Hashes in an Active-Active databases

title: HyperLogLog in Active-Active databases

summary: Information about using JSON with an Active-Active database.

title: JSON in Active-Active databases

summary: Information about using list with an Active-Active database.

title: Lists in Active-Active databases

summary: Information about using sets with an Active-Active database.

title: Sets in Active-Active databases

summary: Information about using sorted sets with an Active-Active database.

title: Sorted sets in Active-Active databases

summary: Information about using streams with an Active-Active database.

title: Streams in Active-Active databases

summary: Information about using strings and bitfields with an Active-Active database.

title: Strings and bitfields in Active-Active databases

title: Content

text:
Active-Active databases use conflict-free replicated data types (CRDTs). From a developer perspective, most supported data types work the same for Active-Active and standard Redis databases. However, a few methods also come with specific requirements in Active-Active databases.

Even though they look identical to standard Redis data types, there are specific rules that govern the handling of
conflicting concurrent writes for each data type.

As conflict handling rules differ between data types, some commands have slightly different requirements in Active-Active databases versus standard Redis databases.

See the following articles for more information.


title: JSON in Active-Active databases

title: Overview

text:
Active-Active databases support JSON data structures.

The design is based on [A Conflict-Free Replicated JSON Datatype](https://arxiv.org/abs/1608.03960) by Kleppmann and Beresford, but the implementation includes some changes. Several [conflict resolution rule](#conflict-resolution-rules) examples were adapted from this paper as well.

title: Prerequisites

text:
To use JSON in an Active-Active database, you must enable JSON during database creation.

Active-Active Redis Cloud databases add JSON by default. See [Create an Active-Active subscription]() in the Redis Cloud documentation for details.

In Redis Software, Active-Active databases created with or upgraded to Redis version 8 or later automatically enable JSON. For earlier Redis versions, see [Create an Active-Active JSON database]() in the Redis Stack and Redis Software documentation for instructions.


title: Lists in Active-Active databases

title: Content

text:
Redis lists are simply lists of strings, sorted by insertion order. It
is possible to add elements to a Redis List that push new elements to
the head (on the left) or to the tail (on the right) of the list. Redis
lists can be used to easily implement queues (using LPUSH and RPOP, for
example) and stacks (using LPUSH and LPOP, for
example).

Lists in Active-Active databases are just the same as regular Redis Lists. See the
following examples to get familiar with Lists' behavior in an
Active-Active database.

Simple Lists
example:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH mylist "hello" |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | LPUSH mylist "world" |
|  t4 | — Sync — | — Sync — |
|  t5 | LRANGE mylist 0 -1 =>"world" "hello" | LRANGE mylist 0 -1 => "world" "hello" |

**Explanation**:
The final list contains both the "world" and "hello" elements, in that
order (Instance 2 observed "hello" when it added
"world").

Example of Lists with Concurrent
Insertions:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | LINSERT L AFTER x y1 |  |
|  t4 |  | LINSERT L AFTER x y2 |
|  t5 | LRANGE L 0 -1 => x y1 | LRANGE L 0 -1 => x y2 |
|  t6 | — Sync — | — Sync — |
|  t7 | LRANGE L 0 -1 => x y1 y2 | LRANGE L 0 -1 => x y1 y2 |

**Explanation**:
Instance 1 added an element y1 after x, and then Instance 2 added element y2 after x.
The final List contains all three elements: x is the first element, after it y1 and then y2.
The Active-Active database resolves the conflict arbitrarily but applies the resolution consistently across all Active-Active database instances.

Example of Deleting a List while Pushing a New
Element:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x |  |
|  t2 | — Sync — | — Sync — |
|  t3 | LRANGE L 0 -1 => x | LRANGE L 0 -1 => x |
|  t4 | LPUSH L y | DEL L |
|  t5 | — Sync — | — Sync — |
|  t6 | LRANGE L 0 -1 => y | LRANGE L 0 -1 => y |

**Explanation**
At t4 - t6, DEL deletes only observed elements. This is why L still
contains y.

Example of Popping Elements from a
List:

|  **Time** | **CRDB Instance 1** | **CRDB Instance 2** |
|  ------: | :------: | :------: |
|  t1 | LPUSH L x y z |  |
|  t2 | — Sync — | — Sync — |
|  t3 |  | RPOP L => x |
|  t4 | — Sync — | — Sync — |
|  t5 | RPOP L => y |  |
|  t6 | — Sync — | — Sync — |
|  t7 | RPOP L => z | RPOP L => z |

**Explanation**:
At t1, the operation pushes elements x, y, z to List L. At t3, the
sequential pops behave as expected from a queue. At t7, the concurrent
pop in both instances might show the same result. The instance was not
able to sync regarding the z removal so, from the point of view of each
instance, z is located in the List and can be popped. After syncing,
both lists are empty.

Be aware of the behavior of Lists in Active-Active databases when using List as a stack
or queue. As seen in the above example, two parallel RPOP operations
performed by two different Active-Active database instances can get the same element in
the case of a concurrent operation. Lists in Active-Active databases guarantee that each
element is POP-ed at least once, but cannot guarantee that each
element is POP-ed only once. Such behavior should be taken into
account when, for example, using Lists in Active-Active databases as building blocks for
inter-process communication systems.

In that case, if the same element cannot be handled twice by the
applications, it's recommended that the POP operations be performed by
one Active-Active database instance, whereas the PUSH operations can be performed by
multiple instances.


title: Sets in Active-Active databases

title: Content

text:
A Redis set is an unordered collection of strings. It is possible to
add, remove, and test for the existence of members with Redis commands.
A Redis set maintains a unique collection of elements. Sets can be great
for maintaining a list of events (click streams), users (in a group
conversation), products (in recommendation lists), engagements (likes,
shares) and so on.

Sets in Active-Active databases behave the same and maintain additional metadata to
achieve an "OR-Set" behavior to handle concurrent conflicting
writes. With the OR-Set behavior, writes across multiple Active-Active database instances
are typically unioned except in cases of conflicts. Conflicting instance
writes can happen when a Active-Active database instance deletes an element while the
other adds the same element. In this case and observed remove rule is
followed. That is, remove can only remove instances it has already seen
and in all other cases element add wins.

Here is an example of an "add wins" case:

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | SADD key1 "a" |  |
|  t2 |  | SADD key1 "b" |
|  t3 | SMEMBERS key1 "a" | SMEMBERS key1 "b" |
|  t4 | — Sync — | — Sync — |
|  t3 | SMEMBERS key1 "a" "b" | SMEMBERS key1 "a" "b" |

Here is an example of an "observed remove" case.

|  **Time** | **CRDB Instance1** | **CRDB Instance2** |
|  ------: | :------: | :------: |
|  t1 | SMEMBERS key1 "a" "b" | SMEMBERS key1 "a" "b" |
|  t2 | SREM key1 "a" | SADD key1 "c" |
|  t3 | SREM key1 "c" |  |
|  t4 | — Sync — | — Sync — |
|  t3 | SMEMBERS key1 "c" "b" | SMEMBERS key1 "c" "b" |