Redis persistence

Summary: How Redis writes data to disk

Overview

Persistence refers to the writing of data to durable storage, such as a solid-state disk (SSD). Redis provides a range of persistence options. These include:

* **RDB** (Redis Database): RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
* **AOF** (Append Only File): AOF persistence logs every write operation received by the server. These operations can then be replayed again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself.
* **No persistence**: You can disable persistence completely. This is sometimes used when caching.
* **RDB + AOF**: You can also combine both AOF and RDB in the same instance.

If you'd rather not think about the tradeoffs between these different persistence strategies, you may want to consider [Redis Software's persistence options](), which can be pre-configured using a UI.

To learn more about how to evaluate your Redis persistence strategy, read on.

RDB advantages

* RDB is a very compact single-file point-in-time representation of your Redis data. RDB files are perfect for backups. For instance you may want to archive your RDB files every hour for the latest 24 hours, and to save an RDB snapshot every day for 30 days. This allows you to easily restore different versions of the data set in case of disasters.
* RDB is very good for disaster recovery, being a single compact file that can be transferred to far data centers, or onto Amazon S3 (possibly encrypted).
* RDB maximizes Redis performances since the only work the Redis parent process needs to do in order to persist is forking a child that will do all the rest. The parent process will never perform disk I/O or alike.
* RDB allows faster restarts with big datasets compared to AOF.
* On replicas, RDB supports [partial resynchronizations after restarts and failovers]().

RDB disadvantages

* RDB is NOT good if you need to minimize the chance of data loss in case Redis stops working (for example after a power outage). You can configure different *save points* where an RDB is produced (for instance after at least five minutes and 100 writes against the data set, you can have multiple save points). However you'll usually create an RDB snapshot every five minutes or more, so in case of Redis stopping working without a correct shutdown for any reason you should be prepared to lose the latest minutes of data.
* RDB needs to fork() often in order to persist on disk using a child process. fork() can be time consuming if the dataset is big, and may result in Redis stopping serving clients for some milliseconds or even for one second if the dataset is very big and the CPU performance is not great. AOF also needs to fork() but less frequently and you can tune how often you want to rewrite your logs without any trade-off on durability.

AOF advantages

* Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second, write performance is still great. fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress, so you can only lose one second worth of writes.
* The AOF log is an append-only log, so there are no seeks, nor corruption problems if there is a power outage. Even if the log ends with a half-written command for some reason (disk full or other reasons) the redis-check-aof tool is able to fix it easily.
* Redis is able to automatically rewrite the AOF in background when it gets too big. The rewrite is completely safe as while Redis continues appending to the old file, a completely new one is produced with the minimal set of operations needed to create the current data set, and once this second file is ready Redis switches the two and starts appending to the new one.
* AOF contains a log of all the operations one after the other in an easy to understand and parse format. You can even easily export an AOF file. For instance even if you've accidentally flushed everything using the [`FLUSHALL`](/commands/flushall) command, as long as no rewrite of the log was performed in the meantime, you can still save your data set just by stopping the server, removing the latest command, and restarting Redis again.

AOF disadvantages

* AOF files are usually bigger than the equivalent RDB files for the same dataset.
* AOF can be slower than RDB depending on the exact fsync policy. In general with fsync set to *every second* performance is still very high, and with fsync disabled it should be exactly as fast as RDB even under high load. Still RDB is able to provide more guarantees about the maximum latency even in the case of a huge write load.

**Redis < 7.0**

* AOF can use a lot of memory if there are writes to the database during a rewrite (these are buffered in memory and written to the new AOF at the end).
* All write commands that arrive during rewrite are written to disk twice.
* Redis could freeze writing and fsyncing these write commands to the new AOF file at the end of the rewrite.

Ok, so what should I use?
---

The general indication you should use both persistence methods is if
you want a degree of data safety comparable to what PostgreSQL can provide you.

If you care a lot about your data, but still can live with a few minutes of
data loss in case of disasters, you can simply use RDB alone.

There are many users using AOF alone, but we discourage it since to have an
RDB snapshot from time to time is a great idea for doing database backups,
for faster restarts, and in the event of bugs in the AOF engine.

The following sections will illustrate a few more details about the two persistence models.

Snapshotting

By default Redis saves snapshots of the dataset on disk, in a binary
file called `dump.rdb`. You can configure Redis to have it save the
dataset every N seconds if there are at least M changes in the dataset,
or you can manually call the [`SAVE`](/commands/save) or [`BGSAVE`](/commands/bgsave) commands.

For example, this configuration will make Redis automatically dump the
dataset to disk every 60 seconds if at least 1000 keys changed:

    save 60 1000

This strategy is known as _snapshotting_.

How it works

Whenever Redis needs to dump the dataset to disk, this is what happens:

* Redis [forks](http://linux.die.net/man/2/fork). We now have a child
and a parent process.

* The child starts to write the dataset to a temporary RDB file.

* When the child is done writing the new RDB file, it replaces the old
one.

This method allows Redis to benefit from copy-on-write semantics.

Append-only file

Snapshotting is not very durable. If your computer running Redis stops,
your power line fails, or you accidentally `kill -9` your instance, the
latest data written to Redis will be lost.  While this may not be a big
deal for some applications, there are use cases for full durability, and
in these cases Redis snapshotting alone is not a viable option.

The _append-only file_ is an alternative, fully-durable strategy for
Redis.  It became available in version 1.1.

You can turn on the AOF in your configuration file:

    appendonly yes

From now on, every time Redis receives a command that changes the
dataset (e.g. [`SET`](/commands/set)) it will append it to the AOF.  When you restart
Redis it will re-play the AOF to rebuild the state.

Since Redis 7.0.0, Redis uses a multi part AOF mechanism.
That is, the original single AOF file is split into base file (at most one) and incremental files (there may be more than one).
The base file represents an initial (RDB or AOF format) snapshot of the data present when the AOF is [rewritten](#log-rewriting).
The incremental files contains incremental changes since the last base AOF file was created. All these files are put in a separate directory and are tracked by a manifest file.

Log rewriting

The AOF gets bigger and bigger as write operations are
performed.  For example, if you are incrementing a counter 100 times,
you'll end up with a single key in your dataset containing the final
value, but 100 entries in your AOF. 99 of those entries are not needed
to rebuild the current state.

The rewrite is completely safe.
While Redis continues appending to the old file,
a completely new one is produced with the minimal set of operations needed to create the current data set,
and once this second file is ready Redis switches the two and starts appending to the new one.

So Redis supports an interesting feature: it is able to rebuild the AOF
in the background without interrupting service to clients. Whenever
you issue a [`BGREWRITEAOF`](/commands/bgrewriteaof), Redis will write the shortest sequence of
commands needed to rebuild the current dataset in memory.  If you're
using the AOF with Redis 2.2 you'll need to run [`BGREWRITEAOF`](/commands/bgrewriteaof) from time to
time. Since Redis 2.4 is able to trigger log rewriting automatically (see the
example configuration file for more information).

Since Redis 7.0.0, when an AOF rewrite is scheduled, the Redis parent process opens a new incremental AOF file to continue writing.
The child process executes the rewrite logic and generates a new base AOF.
Redis will use a temporary manifest file to track the newly generated base file and incremental file.
When they are ready, Redis will perform an atomic replacement operation to make this temporary manifest file take effect.
In order to avoid the problem of creating many incremental files in case of repeated failures and retries of an AOF rewrite,
Redis introduces an AOF rewrite limiting mechanism to ensure that failed AOF rewrites are retried at a slower and slower rate.

How durable is the append only file?

You can configure how many times Redis will
[`fsync`](http://linux.die.net/man/2/fsync) data on disk. There are
three options:

* `appendfsync always`: `fsync` every time new commands are appended to the AOF. Very very slow, very safe. Note that the commands are appended to the AOF after a batch of commands from multiple clients or a pipeline are executed, so it means a single write and a single fsync (before sending the replies).
* `appendfsync everysec`: `fsync` every second. Fast enough (since version 2.4 likely to be as fast as snapshotting), and you may lose 1 second of data if there is a disaster.
* `appendfsync no`: Never `fsync`, just put your data in the hands of the Operating System. The faster and less safe method. Normally Linux will flush data every 30 seconds with this configuration, but it's up to the kernel's exact tuning.

The suggested (and default) policy is to `fsync` every second. It is
both fast and relatively safe. The `always` policy is very slow in
practice, but it supports group commit, so if there are multiple parallel
writes Redis will try to perform a single `fsync` operation.

What should I do if my AOF gets truncated?

It is possible the server crashed while writing the AOF file, or the
volume where the AOF file is stored was full at the time of writing. When this happens the
AOF still contains consistent data representing a given point-in-time version
of the dataset (that may be old up to one second with the default AOF fsync
policy), but the last command in the AOF could be truncated.
The latest major versions of Redis will be able to load the AOF anyway, just
discarding the last non well formed command in the file. In this case the
server will emit a log like the following:

[code example]

You can change the default configuration to force Redis to stop in such
cases if you want, but the default configuration is to continue regardless of
the fact the last command in the file is not well-formed, in order to guarantee
availability after a restart.

Older versions of Redis may not recover, and may require the following steps:

* Make a backup copy of your AOF file.
* Fix the original file using the `redis-check-aof` tool that ships with Redis:

      $ redis-check-aof --fix <filename>

* Optionally use `diff -u` to check what is the difference between two files.
* Restart the server with the fixed file.

What should I do if my AOF gets corrupted?

If the AOF file is not just truncated, but corrupted with invalid byte
sequences in the middle, things are more complex. Redis will complain
at startup and will abort:

[code example]

The best thing to do is to run the `redis-check-aof` utility, initially without
the `--fix` option, then understand the problem, jump to the given
offset in the file, and see if it is possible to manually repair the file:
The AOF uses the same format of the Redis protocol and is quite simple to fix
manually. Otherwise it is possible to let the utility fix the file for us, but
in that case all the AOF portion from the invalid part to the end of the
file may be discarded, leading to a massive amount of data loss if the
corruption happened to be in the initial part of the file.

How it works

Log rewriting uses the same copy-on-write trick already in use for
snapshotting.  This is how it works:

**Redis >= 7.0**

* Redis [forks](http://linux.die.net/man/2/fork), so now we have a child
and a parent process.

* The child starts writing the new base AOF in a temporary file.

* The parent opens a new increments AOF file to continue writing updates.
  If the rewriting fails, the old base and increment files (if there are any) plus this newly opened increment file represent the complete updated dataset,
  so we are safe.

* When the child is done rewriting the base file, the parent gets a signal,
and uses the newly opened increment file and child generated base file to build a temp manifest,
and persist it.

* Profit! Now Redis does an atomic exchange of the manifest files so that the result of this AOF rewrite takes effect. Redis also cleans up the old base file and any unused increment files.

**Redis < 7.0**

* Redis [forks](http://linux.die.net/man/2/fork), so now we have a child
and a parent process.

* The child starts writing the new AOF in a temporary file.

* The parent accumulates all the new changes in an in-memory buffer (but
at the same time it writes the new changes in the old append-only file,
so if the rewriting fails, we are safe).

* When the child is done rewriting the file, the parent gets a signal,
and appends the in-memory buffer at the end of the file generated by the
child.

* Now Redis atomically renames the new file into the old one,
and starts appending new data into the new file.

How I can switch to AOF, if I'm currently using dump.rdb snapshots?

If you want to enable AOF in a server that is currently using RDB snapshots, you need to convert the data by enabling AOF via CONFIG command on the live server first.

**IMPORTANT:** not following this procedure (e.g. just changing the config and restarting the server) can result in data loss!

**Redis >= 2.2**

Preparations:

* Make a backup of your latest dump.rdb file.
* Transfer this backup to a safe place.

Switch to AOF on live database:

* Enable AOF: `redis-cli config set appendonly yes`
* Optionally disable RDB: `redis-cli config set save ""`
* Make sure writes are appended to the append only file correctly.
* **IMPORTANT:** Update your `redis.conf` (potentially through [`CONFIG REWRITE`](/commands/config-rewrite)) and ensure that it matches the configuration above.
  If you forget this step, when you restart the server, the configuration changes will be lost and the server will start again with the old configuration, resulting in a loss of your data.

Next time you restart the server:

* Before restarting the server, wait for AOF rewrite to finish persisting the data.
  You can do that by watching `INFO persistence`, waiting for `aof_rewrite_in_progress` and `aof_rewrite_scheduled` to be `0`, and validating that `aof_last_bgrewrite_status` is `ok`.
* After restarting the server, check that your database contains the same number of keys it contained previously.

**Redis 2.0**

* Make a backup of your latest dump.rdb file.
* Transfer this backup into a safe place.
* Stop all the writes against the database!
* Issue a `redis-cli BGREWRITEAOF`. This will create the append only file.
* Stop the server when Redis finished generating the AOF dump.
* Edit redis.conf end enable append only file persistence.
* Restart the server.
* Make sure that your database contains the same number of keys it contained before the switch.
* Make sure that writes are appended to the append only file correctly.

Interactions between AOF and RDB persistence

Redis >= 2.4 makes sure to avoid triggering an AOF rewrite when an RDB
snapshotting operation is already in progress, or allowing a [`BGSAVE`](/commands/bgsave) while the
AOF rewrite is in progress. This prevents two Redis background processes
from doing heavy disk I/O at the same time.

When snapshotting is in progress and the user explicitly requests a log
rewrite operation using [`BGREWRITEAOF`](/commands/bgrewriteaof) the server will reply with an OK
status code telling the user the operation is scheduled, and the rewrite
will start once the snapshotting is completed.

In the case both AOF and RDB persistence are enabled and Redis restarts the
AOF file will be used to reconstruct the original dataset since it is
guaranteed to be the most complete.

Backing up Redis data

Before starting this section, make sure to read the following sentence: **Make Sure to Backup Your Database**. Disks break, instances in the cloud disappear, and so forth: no backups means huge risk of data disappearing into /dev/null.

Redis is very data backup friendly since you can copy RDB files while the
database is running: the RDB is never modified once produced, and while it
gets produced it uses a temporary name and is renamed into its final destination
atomically using rename(2) only when the new snapshot is complete.

This means that copying the RDB file is completely safe while the server is
running. This is what we suggest:

* Create a cron job in your server creating hourly snapshots of the RDB file in one directory, and daily snapshots in a different directory.
* Every time the cron script runs, make sure to call the `find` command to make sure too old snapshots are deleted: for instance you can take hourly snapshots for the latest 48 hours, and daily snapshots for one or two months. Make sure to name the snapshots with date and time information.
* At least one time every day make sure to transfer an RDB snapshot *outside your data center* or at least *outside the physical machine* running your Redis instance.

Backing up AOF persistence

If you run a Redis instance with only AOF persistence enabled, you can still perform backups.
Since Redis 7.0.0, AOF files are split into multiple files which reside in a single directory determined by the `appenddirname` configuration.
During normal operation all you need to do is copy/tar the files in this directory to achieve a backup. However, if this is done during a [rewrite](#log-rewriting), you might end up with an invalid backup.
To work around this you must disable AOF rewrites during the backup:

1. Turn off automatic rewrites with

   [`CONFIG SET`](/commands/config-set) `auto-aof-rewrite-percentage 0`

   Make sure you don't manually start a rewrite (using [`BGREWRITEAOF`](/commands/bgrewriteaof)) during this time.
2. Check there's no current rewrite in progress using

   [`INFO`](/commands/info) `persistence`

   and verifying `aof_rewrite_in_progress` is 0. If it's 1, then you'll need to wait for the rewrite to complete.
3. Now you can safely copy the files in the `appenddirname` directory.
4. Re-enable rewrites when done:

   [`CONFIG SET`](/commands/config-set) `auto-aof-rewrite-percentage <prev-value>`

**Note:** If you want to minimize the time AOF rewrites are disabled you may create hard links to the files in `appenddirname` (in step 3 above) and then re-enable rewrites (step 4) after the hard links are created.
Now you can copy/tar the hardlinks and delete them when done. This works because Redis guarantees that it
only appends to files in this directory, or completely replaces them if necessary, so the content should be
consistent at any given point in time.

**Note:** If you want to handle the case of the server being restarted during the backup and make sure no rewrite will automatically start after the restart you can change step 1 above to also persist the updated configuration via [`CONFIG REWRITE`](/commands/config-rewrite).
Just make sure to re-enable automatic rewrites when done (step 4) and persist it with another [`CONFIG REWRITE`](/commands/config-rewrite).

Prior to version 7.0.0 backing up the AOF file can be done simply by copying the aof file (like backing up the RDB snapshot). The file may lack the final part
but Redis will still be able to load it (see the previous sections about [truncated AOF files](#what-should-i-do-if-my-aof-gets-truncated)).

Disaster recovery

Disaster recovery in the context of Redis is basically the same story as
backups, plus the ability to transfer those backups in many different external
data centers. This way data is secured even in the case of some catastrophic
event affecting the main data center where Redis is running and producing its
snapshots.

We'll review the most interesting disaster recovery techniques
that don't have too high costs.

* Amazon S3 and other similar services are a good way for implementing your disaster recovery system. Simply transfer your daily or hourly RDB snapshot to S3 in an encrypted form. You can encrypt your data using `gpg -c` (in symmetric encryption mode). Make sure to store your password in many different safe places (for instance give a copy to the most important people of your organization). It is recommended to use multiple storage services for improved data safety.
* Transfer your snapshots using SCP (part of SSH) to far servers. This is a fairly simple and safe route: get a small VPS in a place that is very far from you, install ssh there, and generate a ssh client key without passphrase, then add it in the `authorized_keys` file of your small VPS. You are ready to transfer backups in an automated fashion. Get at least two VPS in two different providers
for best results.

It is important to understand that this system can easily fail if not
implemented in the right way. At least, make absolutely sure that after the
transfer is completed you are able to verify the file size (that should match
the one of the file you copied) and possibly the SHA1 digest, if you are using
a VPS.

You also need some kind of independent alert system if the transfer of fresh
backups is not working for some reason.

* Reading RDB preamble from AOF file...
* Reading the remaining AOF tail...
# !!! Warning: short read while loading the AOF file !!!
# !!! Truncating the AOF at offset 439 !!!
# AOF loaded anyway because aof-load-truncated is enabled

* Reading the remaining AOF tail...
# Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>


Redis replication

Summary: How Redis supports high availability and failover with replication

Overview

At the base of Redis replication (excluding the high availability features provided as an additional layer by Redis Cluster or Redis Sentinel) there is a *leader follower* (master-replica) replication that is simple to use and configure. It allows replica Redis instances to be exact copies of master instances. The replica will automatically reconnect to the master every time the link breaks, and will attempt to be an exact copy of it *regardless* of what happens to the master.

This system works using three main mechanisms:

1. When a master and a replica instance are well-connected, the master keeps the replica updated by sending a stream of commands to the replica to replicate the effects on the dataset happening on the master side due to: client writes, keys expired or evicted, any other action changing the master dataset.
2. When the link between the master and the replica breaks, for network issues or because a timeout is sensed in the master or the replica, the replica reconnects and attempts to proceed with a partial resynchronization: it means that it will try to just obtain the part of the stream of commands it missed during the disconnection.
3. When a partial resynchronization is not possible, the replica will ask for a full resynchronization. This will involve a more complex process in which the master needs to create a snapshot of all its data, send it to the replica, and then continue sending the stream of commands as the dataset changes.

Redis uses by default asynchronous replication, which being low latency and
high performance, is the natural replication mode for the vast majority of Redis
use cases. However, Redis replicas asynchronously acknowledge the amount of data
they receive periodically with the master. So the master does not wait every time
for a command to be processed by the replicas, however it knows, if needed, what
replica already processed what command. This allows having optional synchronous replication.

Synchronous replication of certain data can be requested by the clients using
the [`WAIT`](/commands/wait) command. However [`WAIT`](/commands/wait) is only able to ensure there are the
specified number of acknowledged copies in the other Redis instances, it does not
turn a set of Redis instances into a CP system with strong consistency: acknowledged
writes can still be lost during a failover, depending on the exact configuration
of the Redis persistence. However, [WAIT]() dramatically reduces the probability of losing a write after a failure event to specific hard-to-trigger failure modes.

You can check the Redis Sentinel or Redis Cluster documentation for more information
about high availability and failover. The rest of this document mainly describes the basic characteristics of Redis basic replication.

Important facts about Redis replication

* Redis uses asynchronous replication, with asynchronous replica-to-master acknowledges of the amount of data processed.
* A master can have multiple replicas.
* Replicas are able to accept connections from other replicas. Aside from connecting a number of replicas to the same master, replicas can also be connected to other replicas in a cascading-like structure. Since Redis 4.0, all the sub-replicas will receive exactly the same replication stream from the master.
* Redis replication is non-blocking on the master side. This means that the master will continue to handle queries when one or more replicas perform the initial synchronization or a partial resynchronization.
* Replication is also largely non-blocking on the replica side. While the replica is performing the initial synchronization, it can handle queries using the old version of the dataset, assuming you configured Redis to do so in redis.conf.  Otherwise, you can configure Redis replicas to return an error to clients if the replication stream is down. However, after the initial sync, the old dataset must be deleted and the new one must be loaded. The replica will block incoming connections during this brief window (that can be as long as many seconds for very large datasets). Since Redis 4.0 you can configure Redis so that the deletion of the old data set happens in a different thread, however loading the new initial dataset will still happen in the main thread and block the replica.
* Replication can be used both for scalability, to have multiple replicas for read-only queries (for example, slow O(N) operations can be offloaded to replicas), or simply for improving data safety and high availability.
* You can use replication to avoid the cost of having the master writing the full dataset to disk: a typical technique involves configuring your master `redis.conf` to avoid persisting to disk at all, then connect a replica configured to save from time to time, or with AOF enabled. However, this setup must be handled with care, since a restarting master will start with an empty dataset: if the replica tries to sync with it, the replica will be emptied as well.

Safety of replication when master has persistence turned off

In setups where Redis replication is used, it is strongly advised to have
persistence turned on in the master and in the replicas. When this is not possible,
for example because of latency concerns due to very slow disks, instances should
be configured to **avoid restarting automatically** after a reboot.

To better understand why masters with persistence turned off configured to
auto restart are dangerous, check the following failure mode where data
is wiped from the master and all its replicas:

1. We have a setup with node A acting as master, with persistence turned down, and nodes B and C replicating from node A.
2. Node A crashes, however it has some auto-restart system, that restarts the process. However since persistence is turned off, the node restarts with an empty data set.
3. Nodes B and C will replicate from node A, which is empty, so they'll effectively destroy their copy of the data.

When Redis Sentinel is used for high availability, also turning off persistence
on the master, together with auto restart of the process, is dangerous. For example the master can restart fast enough for Sentinel to not detect a failure, so that the failure mode described above happens.

Every time data safety is important, and replication is used with master configured without persistence, auto restart of instances should be disabled.

How Redis replication works

Every Redis master has a replication ID: it is a large pseudo random string
that marks a given history of the dataset. Each master also takes an offset that
increments for every byte of replication stream that it is produced to be
sent to replicas, to update the state of the replicas with the new changes
modifying the dataset. The replication offset is incremented even if no replica
is actually connected, so basically every given pair of:

    Replication ID, offset

Identifies an exact version of the dataset of a master.

When replicas connect to masters, they use the [`PSYNC`](/commands/psync) command to send
their old master replication ID and the offsets they processed so far. This way
the master can send just the incremental part needed. However if there is not
enough *backlog* in the master buffers, or if the replica is referring to an
history (replication ID) which is no longer known, then a full resynchronization
happens: in this case the replica will get a full copy of the dataset, from scratch.

This is how a full synchronization works in more details:

The master starts a background saving process to produce an RDB file. At the same time it starts to buffer all new write commands received from the clients. When the background saving is complete, the master transfers the database file to the replica, which saves it on disk, and then loads it into memory. The master will then send all buffered commands to the replica. This is done as a stream of commands and is in the same format of the Redis protocol itself.

You can try it yourself via telnet. Connect to the Redis port while the
server is doing some work and issue the [`SYNC`](/commands/sync) command. You'll see a bulk
transfer and then every command received by the master will be re-issued
in the telnet session. Actually [`SYNC`](/commands/sync) is an old protocol no longer used by
newer Redis instances, but is still there for backward compatibility: it does
not allow partial resynchronizations, so now [`PSYNC`](/commands/psync) is used instead.

As already said, replicas are able to automatically reconnect when the master-replica link goes down for some reason. If the master receives multiple concurrent replica synchronization requests, it performs a single background save in to serve all of them.

Replication ID explained

In the previous section we said that if two instances have the same replication
ID and replication offset, they have exactly the same data. However it is useful
to understand what exactly is the replication ID, and why instances have actually
two replication IDs: the main ID and the secondary ID.

A replication ID basically marks a given *history* of the data set. Every time
an instance restarts from scratch as a master, or a replica is promoted to master,
a new replication ID is generated for this instance. The replicas connected to
a master will inherit its replication ID after the handshake. So two instances
with the same ID are related by the fact that they hold the same data, but
potentially at a different time. It is the offset that works as a logical time
to understand, for a given history (replication ID), who holds the most updated
data set.

For instance, if two instances A and B have the same replication ID, but one
with offset 1000 and one with offset 1023, it means that the first lacks certain
commands applied to the data set. It also means that A, by applying just a few
commands, may reach exactly the same state of B.

The reason why Redis instances have two replication IDs is because of replicas
that are promoted to masters. After a failover, the promoted replica requires
to still remember what was its past replication ID, because such replication ID
was the one of the former master. In this way, when other replicas will sync
with the new master, they will try to perform a partial resynchronization using the
old master replication ID. This will work as expected, because when the replica
is promoted to master it sets its secondary ID to its main ID, remembering what
was the offset when this ID switch happened. Later it will select a new random
replication ID, because a new history begins. When handling the new replicas
connecting, the master will match their IDs and offsets both with the current
ID and the secondary ID (up to a given offset, for safety). In short this means
that after a failover, replicas connecting to the newly promoted master don't have
to perform a full sync.

In case you wonder why a replica promoted to master needs to change its
replication ID after a failover: it is possible that the old master is still
working as a master because of some network partition: retaining the same
replication ID would violate the fact that the same ID and same offset of any
two random instances mean they have the same data set.

Diskless replication

Normally a full resynchronization requires creating an RDB file on disk,
then reloading the same RDB from disk to feed the replicas with the data.

With slow disks this can be a very stressing operation for the master.
Redis version 2.8.18 is the first version to have support for diskless
replication. In this setup the child process directly sends the
RDB over the wire to replicas, without using the disk as intermediate storage.

Configuration

To configure basic Redis replication is trivial: just add the following line to the replica configuration file:

    replicaof 192.168.1.1 6379

Of course you need to replace 192.168.1.1 6379 with your master IP address (or
hostname) and port. Alternatively, you can call the [`REPLICAOF`](/commands/replicaof) command and the
master host will start a sync with the replica.

There are also a few parameters for tuning the replication backlog taken
in memory by the master to perform the partial resynchronization. See the example
`redis.conf` shipped with the Redis distribution for more information.

Diskless replication can be enabled using the `repl-diskless-sync` configuration
parameter. The delay to start the transfer to wait for more replicas to
arrive after the first one is controlled by the `repl-diskless-sync-delay`
parameter. Please refer to the example `redis.conf` file in the Redis distribution
for more details.

Read-only replica

Since Redis 2.6, replicas support a read-only mode that is enabled by default.
This behavior is controlled by the `replica-read-only` option in the redis.conf file, and can be enabled and disabled at runtime using [`CONFIG SET`](/commands/config-set).

Read-only replicas will reject all write commands, so that it is not possible to write to a replica because of a mistake. This does not mean that the feature is intended to expose a replica instance to the internet or more generally to a network where untrusted clients exist, because administrative commands like [`DEBUG`](/commands/debug) or [`CONFIG`](/commands/config) are still enabled. The [Security]() page describes how to secure a Redis instance.

You may wonder why it is possible to revert the read-only setting
and have replica instances that can be targeted by write operations.
The answer is that writable replicas exist only for historical reasons.
Using writable replicas can result in inconsistency between the master and the replica, so it is not recommended to use writable replicas.
To understand in which situations this can be a problem, we need to understand how replication works.
Changes on the master is replicated by propagating regular Redis commands to the replica.
When a key expires on the master, this is propagated as a DEL command.
If a key which exists on the master but is deleted, expired or has a different type on the replica compared to the master will react differently to commands like DEL, INCR or RPOP propagated from the master than intended.
The propagated command may fail on the replica or result in a different outcome.
To minimize the risks (if you insist on using writable replicas) we suggest you follow these recommendations:

* Don't write to keys in a writable replica that are also used on the master.
  (This can be hard to guarantee if you don't have control over all the clients that write to the master.)

* Don't configure an instance as a writable replica as an intermediary step when upgrading a set of instances in a running system.
  In general, don't configure an instance as a writable replica if it can ever be promoted to a master if you want to guarantee data consistency.

Historically, there were some use cases that were considered legitimate for writable replicas.
As of version 7.0, these use cases are now all obsolete and the same can be achieved by other means.
For example:

* Computing slow Set or Sorted set operations and storing the result in temporary local keys using commands like [`SUNIONSTORE`](/commands/sunionstore) and [`ZINTERSTORE`](/commands/zinterstore).
  Instead, use commands that return the result without storing it, such as [`SUNION`](/commands/sunion) and [`ZINTER`](/commands/zinter).

* Using the [`SORT`](/commands/sort) command (which is not considered a read-only command because of the optional STORE option and therefore cannot be used on a read-only replica).
  Instead, use [`SORT_RO`](/commands/sort_ro), which is a read-only command.

* Using [`EVAL`](/commands/eval) and [`EVALSHA`](/commands/evalsha) are also not considered read-only commands, because the Lua script may call write commands.
  Instead, use [`EVAL_RO`](/commands/eval_ro) and [`EVALSHA_RO`](/commands/evalsha_ro) where the Lua script can only call read-only commands.

While writes to a replica will be discarded if the replica and the master resync or if the replica is restarted, there is no guarantee that they will sync automatically.

Before version 4.0, writable replicas were incapable of expiring keys with a time to live set.
This means that if you use [`EXPIRE`](/commands/expire) or other commands that set a maximum TTL for a key, the key will leak, and while you may no longer see it while accessing it with read commands, you will see it in the count of keys and it will still use memory.
Redis 4.0 RC3 and greater versions are able to evict keys with TTL as masters do, with the exceptions of keys written in DB numbers greater than 63 (but by default Redis instances only have 16 databases).
Note though that even in versions greater than 4.0, using [`EXPIRE`](/commands/expire) on a key that could ever exists on the master can cause inconsistency between the replica and the master.

Also note that since Redis 4.0 replica writes are only local, and are not propagated to sub-replicas attached to the instance. Sub-replicas instead will always receive the replication stream identical to the one sent by the top-level master to the intermediate replicas. So for example in the following setup:

    A ---> B ---> C

Even if `B` is writable, C will not see `B` writes and will instead have identical dataset as the master instance `A`.

Setting a replica to authenticate to a master

If your master has a password via `requirepass`, it's trivial to configure the
replica to use that password in all sync operations.

To do it on a running instance, use `redis-cli` and type:

    config set masterauth <password>

To set it permanently, add this to your config file:

    masterauth <password>

Allow writes only with N attached replicas

Starting with Redis 2.8, you can configure a Redis master to
accept write queries only if at least N replicas are currently connected to the
master.

However, because Redis uses asynchronous replication it is not possible to ensure
the replica actually received a given write, so there is always a window for data
loss.

This is how the feature works:

* Redis replicas ping the master every second, acknowledging the amount of replication stream processed.
* Redis masters will remember the last time it received a ping from every replica.
* The user can configure a minimum number of replicas that have a lag not greater than a maximum number of seconds.

If there are at least N replicas, with a lag less than M seconds, then the write will be accepted.

You may think of it as a best effort data safety mechanism, where consistency is not ensured for a given write, but at least the time window for data loss is restricted to a given number of seconds. In general bound data loss is better than unbound one.

If the conditions are not met, the master will instead reply with an error and the write will not be accepted.

There are two configuration parameters for this feature:

* min-replicas-to-write `<number of replicas>`
* min-replicas-max-lag `<number of seconds>`

For more information, please check the example `redis.conf` file shipped with the
Redis source distribution.

How Redis replication deals with expires on keys

Redis expires allow keys to have a limited time to live (TTL). Such a feature depends
on the ability of an instance to count the time, however Redis replicas correctly
replicate keys with expires, even when such keys are altered using Lua
scripts.

To implement such a feature Redis cannot rely on the ability of the master and
replica to have synced clocks, since this is a problem that cannot be solved
and would result in race conditions and diverging data sets, so Redis
uses three main techniques to make the replication of expired keys
able to work:

1. Replicas don't expire keys, instead they wait for masters to expire the keys. When a master expires a key (or evict it because of LRU), it synthesizes a [`DEL`](/commands/del) command which is transmitted to all the replicas.
2. However because of master-driven expire, sometimes replicas may still have in memory keys that are already logically expired, since the master was not able to provide the [`DEL`](/commands/del) command in time. To deal with that the replica uses its logical clock to report that a key does not exist **only for read operations** that don't violate the consistency of the data set (as new commands from the master will arrive). In this way replicas avoid reporting logically expired keys that are still existing. In practical terms, an HTML fragments cache that uses replicas to scale will avoid returning items that are already older than the desired time to live.
3. During Lua scripts executions no key expiries are performed. As a Lua script runs, conceptually the time in the master is frozen, so that a given key will either exist or not for all the time the script runs. This prevents keys expiring in the middle of a script, and is needed to send the same script to the replica in a way that is guaranteed to have the same effects in the data set.

Once a replica is promoted to a master it will start to expire keys independently, and will not require any help from its old master.

Configuring replication in Docker and NAT

When Docker, or other types of containers using port forwarding, or Network Address Translation is used, Redis replication needs some extra care, especially when using Redis Sentinel or other systems where the master [`INFO`](/commands/info) or [`ROLE`](/commands/role) commands output is scanned to discover replicas' addresses.

The problem is that the [`ROLE`](/commands/role) command, and the replication section of
the [`INFO`](/commands/info) output, when issued into a master instance, will show replicas
as having the IP address they use to connect to the master, which, in
environments using NAT may be different compared to the logical address of the
replica instance (the one that clients should use to connect to replicas).

Similarly the replicas will be listed with the listening port configured
into `redis.conf`, that may be different from the forwarded port in case
the port is remapped.

To fix both issues, it is possible, since Redis 3.2.2, to force
a replica to announce an arbitrary pair of IP and port to the master.
The two configurations directives to use are:

    replica-announce-ip 5.5.5.5
    replica-announce-port 1234

And are documented in the example `redis.conf` of recent Redis distributions.

The INFO and ROLE command

There are two Redis commands that provide a lot of information on the current
replication parameters of master and replica instances. One is [`INFO`](/commands/info). If the
command is called with the `replication` argument as `INFO replication` only
information relevant to the replication are displayed. Another more
computer-friendly command is [`ROLE`](/commands/role), that provides the replication status of
masters and replicas together with their replication offsets, list of connected
replicas and so forth.

Partial sync after restarts and failovers

Since Redis 4.0, when an instance is promoted to master after a failover,
it will still be able to perform a partial resynchronization with the replicas
of the old master. To do so, the replica remembers the old replication ID and
offset of its former master, so can provide part of the backlog to the connecting
replicas even if they ask for the old replication ID.

However the new replication ID of the promoted replica will be different, since it
constitutes a different history of the data set. For example, the master can
return available and can continue accepting writes for some time, so using the
same replication ID in the promoted replica would violate the rule that a
replication ID and offset pair identifies only a single data set.

Moreover, replicas - when powered off gently and restarted - are able to store
in the `RDB` file the information needed to resync with their
master. This is useful in case of upgrades. When this is needed, it is better to
use the [`SHUTDOWN`](/commands/shutdown) command in order to perform a `save & quit` operation on the
replica.

It is not possible to partially sync a replica that restarted via the
AOF file. However the instance may be turned to RDB persistence before shutting
down it, than can be restarted, and finally AOF can be enabled again.

`Maxmemory` on replicas

By default, a replica will ignore `maxmemory` (unless it is promoted to master after a failover or manually).
It means that the eviction of keys will be handled by the master, sending the DEL commands to the replica as keys evict in the master side.

This behavior ensures that masters and replicas stay consistent, which is usually what you want.
However, if your replica is writable, or you want the replica to have a different memory setting, and you are sure all the writes performed to the replica are idempotent, then you may change this default (but be sure to understand what you are doing).

Note that since the replica by default does not evict, it may end up using more memory than what is set via `maxmemory` (since there are certain buffers that may be larger on the replica, or data structures may sometimes take more memory and so forth).
Make sure you monitor your replicas, and make sure they have enough memory to never hit a real out-of-memory condition before the master hits the configured `maxmemory` setting.

To change this behavior, you can allow a replica to not ignore the `maxmemory`. The configuration directives to use is:

    replica-ignore-maxmemory no


Scale with Redis Cluster

Summary: Horizontal scaling with Redis Cluster

Overview

Redis scales horizontally with a deployment topology called Redis Cluster.
This topic will teach you how to set up, test, and operate Redis Cluster in production.
You will learn about the availability and consistency characteristics of Redis Cluster from the end user's point of view.

If you plan to run a production Redis Cluster deployment or want to understand better how Redis Cluster works internally, consult the [Redis Cluster specification](). To learn how Redis Enterprise handles scaling, see [Linear Scaling with Redis Enterprise](https://redis.com/redis-enterprise/technology/linear-scaling-redis-enterprise/).

Redis Cluster 101

Redis Cluster provides a way to run a Redis installation where data is automatically sharded across multiple Redis nodes.
Redis Cluster also provides some degree of availability during partitions—in practical terms, the ability to continue operations when some nodes fail or are unable to communicate.
However, the cluster will become unavailable in the event of larger failures (for example, when the majority of masters are unavailable).

So, with Redis Cluster, you get the ability to:

* Automatically split your dataset among multiple nodes.
* Continue operations when a subset of the nodes are experiencing failures or are unable to communicate with the rest of the cluster.

#### Redis Cluster TCP ports

Every Redis Cluster node requires two open TCP connections: a Redis TCP port used to serve clients, e.g., 6379, and second port known as the _cluster bus port_.
By default, the cluster bus port is set by adding 10000 to the data port (e.g., 16379); however, you can override this in the `cluster-port` configuration.

Cluster bus is a node-to-node communication channel that uses a binary protocol, which is more suited to exchanging information between nodes due to
little bandwidth and processing time.
Nodes use the cluster bus for failure detection, configuration updates, failover authorization, and so forth.
Clients should never try to communicate with the cluster bus port, but rather use the Redis command port.
However, make sure you open both ports in your firewall, otherwise Redis cluster nodes won't be able to communicate.

For a Redis Cluster to work properly you need, for each node:

1. The client communication port (usually 6379) used to communicate with clients and be open to all the clients that need to reach the cluster, plus all the other cluster nodes that use the client port for key migrations.
2. The cluster bus port must be reachable from all the other cluster nodes.

If you don't open both TCP ports, your cluster will not work as expected.

#### Redis Cluster and Docker

Currently, Redis Cluster does not support NATted environments and in general
environments where IP addresses or TCP ports are remapped.

Docker uses a technique called _port mapping_: programs running inside Docker containers may be exposed with a different port compared to the one the program believes to be using.
This is useful for running multiple containers using the same ports, at the same time, in the same server.

To make Docker compatible with Redis Cluster, you need to use Docker's _host networking mode_.
Please see the `--net=host` option in the [Docker documentation](https://docs.docker.com/engine/userguide/networking/dockernetworks/) for more information.

#### Redis Cluster data sharding

Redis Cluster does not use consistent hashing, but a different form of sharding
where every key is conceptually part of what we call a **hash slot**.

There are 16384 hash slots in Redis Cluster, and to compute the hash
slot for a given key, we simply take the CRC16 of the key modulo
16384.

Every node in a Redis Cluster is responsible for a subset of the hash slots,
so, for example, you may have a cluster with 3 nodes, where:

* Node A contains hash slots from 0 to 5500.
* Node B contains hash slots from 5501 to 11000.
* Node C contains hash slots from 11001 to 16383.

This makes it easy to add and remove cluster nodes. For example, if
I want to add a new node D, I need to move some hash slots from nodes A, B, C
to D. Similarly, if I want to remove node A from the cluster, I can just
move the hash slots served by A to B and C. Once node A is empty,
I can remove it from the cluster completely.

Moving hash slots from a node to another does not require stopping
any operations; therefore, adding and removing nodes, or changing the percentage of hash slots held by a node, requires no downtime.

Redis Cluster supports multiple key operations as long as all of the keys involved in a single command execution (or whole transaction, or Lua script
execution) belong to the same hash slot. The user can force multiple keys
to be part of the same hash slot by using a feature called *hash tags*.

Hash tags are documented in the Redis Cluster specification, but the gist is
that if there is a substring between {} brackets in a key, only what is
inside the string is hashed. For example, the keys `user:{123}:profile` and `user:{123}:account` are guaranteed to be in the same hash slot because they share the same hash tag. As a result, you can operate on these two keys in the same multi-key operation.

#### Redis Cluster master-replica model

To remain available when a subset of master nodes are failing or are
not able to communicate with the majority of nodes, Redis Cluster uses a
master-replica model where every hash slot has from 1 (the master itself) to N
replicas (N-1 additional replica nodes).

In our example cluster with nodes A, B, C, if node B fails the cluster is not
able to continue, since we no longer have a way to serve hash slots in the
range 5501-11000.

However, when the cluster is created (or at a later time), we add a replica
node to every master, so that the final cluster is composed of A, B, C
that are master nodes, and A1, B1, C1 that are replica nodes.
This way, the system can continue if node B fails.

Node B1 replicates B, and B fails, the cluster will promote node B1 as the new
master and will continue to operate correctly.

However, note that if nodes B and B1 fail at the same time, Redis Cluster will not be able to continue to operate.

#### Redis Cluster consistency guarantees

Redis Cluster does not guarantee **strong consistency**. In practical
terms this means that under certain conditions it is possible that Redis
Cluster will lose writes that were acknowledged by the system to the client.

The first reason why Redis Cluster can lose writes is because it uses
asynchronous replication. This means that during writes the following
happens:

* Your client writes to the master B.
* The master B replies OK to your client.
* The master B propagates the write to its replicas B1, B2 and B3.

As you can see, B does not wait for an acknowledgement from B1, B2, B3 before
replying to the client, since this would be a prohibitive latency penalty
for Redis, so if your client writes something, B acknowledges the write,
but crashes before being able to send the write to its replicas, one of the
replicas (that did not receive the write) can be promoted to master, losing
the write forever.

This is very similar to what happens with most databases that are
configured to flush data to disk every second, so it is a scenario you
are already able to reason about because of past experiences with traditional
database systems not involving distributed systems. Similarly you can
improve consistency by forcing the database to flush data to disk before
replying to the client, but this usually results in prohibitively low
performance. That would be the equivalent of synchronous replication in
the case of Redis Cluster.

Basically, there is a trade-off to be made between performance and consistency.

Redis Cluster has support for synchronous writes when absolutely needed,
implemented via the [`WAIT`](/commands/wait) command. This makes losing writes a lot less
likely. However, note that Redis Cluster does not implement strong consistency
even when synchronous replication is used: it is always possible, under more
complex failure scenarios, that a replica that was not able to receive the write
will be elected as master.

There is another notable scenario where Redis Cluster will lose writes, that
happens during a network partition where a client is isolated with a minority
of instances including at least a master.

Take as an example our 6 nodes cluster composed of A, B, C, A1, B1, C1,
with 3 masters and 3 replicas. There is also a client, that we will call Z1.

After a partition occurs, it is possible that in one side of the
partition we have A, C, A1, B1, C1, and in the other side we have B and Z1.

Z1 is still able to write to B, which will accept its writes. If the
partition heals in a very short time, the cluster will continue normally.
However, if the partition lasts enough time for B1 to be promoted to master
on the majority side of the partition, the writes that Z1 has sent to B
in the meantime will be lost.

There is a **maximum window** to the amount of writes Z1 will be able
to send to B: if enough time has elapsed for the majority side of the
partition to elect a replica as master, every master node in the minority
side will have stopped accepting writes.

This amount of time is a very important configuration directive of Redis
Cluster, and is called the **node timeout**.

After node timeout has elapsed, a master node is considered to be failing,
and can be replaced by one of its replicas.
Similarly, after node timeout has elapsed without a master node to be able
to sense the majority of the other master nodes, it enters an error state
and stops accepting writes.

Redis Cluster configuration parameters

We are about to create an example cluster deployment.
Before we continue, let's introduce the configuration parameters that Redis Cluster introduces
in the `redis.conf` file.

* **cluster-enabled `<yes/no>`**: If yes, enables Redis Cluster support in a specific Redis instance. Otherwise the instance starts as a standalone instance as usual.
* **cluster-config-file `<filename>`**: Note that despite the name of this option, this is not a user editable configuration file, but the file where a Redis Cluster node automatically persists the cluster configuration (the state, basically) every time there is a change, in order to be able to re-read it at startup. The file lists things like the other nodes in the cluster, their state, persistent variables, and so forth. Often this file is rewritten and flushed on disk as a result of some message reception.
* **cluster-node-timeout `<milliseconds>`**: The maximum amount of time a Redis Cluster node can be unavailable, without it being considered as failing. If a master node is not reachable for more than the specified amount of time, it will be failed over by its replicas. This parameter controls other important things in Redis Cluster. Notably, every node that can't reach the majority of master nodes for the specified amount of time, will stop accepting queries.
* **cluster-slave-validity-factor `<factor>`**: If set to zero, a replica will always consider itself valid, and will therefore always try to failover a master, regardless of the amount of time the link between the master and the replica remained disconnected. If the value is positive, a maximum disconnection time is calculated as the *node timeout* value multiplied by the factor provided with this option, and if the node is a replica, it will not try to start a failover if the master link was disconnected for more than the specified amount of time. For example, if the node timeout is set to 5 seconds and the validity factor is set to 10, a replica disconnected from the master for more than 50 seconds will not try to failover its master. Note that any value different than zero may result in Redis Cluster being unavailable after a master failure if there is no replica that is able to failover it. In that case the cluster will return to being available only when the original master rejoins the cluster.
* **cluster-migration-barrier `<count>`**: Minimum number of replicas a master will remain connected with, for another replica to migrate to a master which is no longer covered by any replica. See the appropriate section about replica migration in this tutorial for more information.
* **cluster-require-full-coverage `<yes/no>`**: If this is set to yes, as it is by default, the cluster stops accepting writes if some percentage of the key space is not covered by any node. If the option is set to no, the cluster will still serve queries even if only requests about a subset of keys can be processed.
* **cluster-allow-reads-when-down `<yes/no>`**: If this is set to no, as it is by default, a node in a Redis Cluster will stop serving all traffic when the cluster is marked as failed, either when a node can't reach a quorum of masters or when full coverage is not met. This prevents reading potentially inconsistent data from a node that is unaware of changes in the cluster. This option can be set to yes to allow reads from a node during the fail state, which is useful for applications that want to prioritize read availability but still want to prevent inconsistent writes. It can also be used for when using Redis Cluster with only one or two shards, as it allows the nodes to continue serving writes when a master fails but automatic failover is impossible.

Create and use a Redis Cluster

To create and use a Redis Cluster, follow these steps:

* [Create a Redis Cluster](#create-a-redis-cluster)
* [Interact with the cluster](#interact-with-the-cluster)
* [Write an example app with redis-rb-cluster](#write-an-example-app-with-redis-rb-cluster)
* [Reshard the cluster](#reshard-the-cluster)
* [A more interesting example application](#a-more-interesting-example-application)
* [Test the failover](#test-the-failover)
* [Manual failover](#manual-failover)
* [Add a new node](#add-a-new-node)
* [Remove a node](#remove-a-node)
* [Replica migration](#replica-migration)
* [Upgrade nodes in a Redis Cluster](#upgrade-nodes-in-a-redis-cluster)
* [Migrate to Redis Cluster](#migrate-to-redis-cluster)

But, first, familiarize yourself with the requirements for creating a cluster.

#### Requirements to create a Redis Cluster

To create a cluster, the first thing you need is to have a few empty Redis instances running in _cluster mode_.

At minimum, set the following directives in the `redis.conf` file:

[code example]

To enable cluster mode, set the `cluster-enabled` directive to `yes`.
Every instance also contains the path of a file where the
configuration for this node is stored, which by default is `nodes.conf`.
This file is never touched by humans; it is simply generated at startup
by the Redis Cluster instances, and updated every time it is needed.

Note that the **minimal cluster** that works as expected must contain
at least three master nodes. For deployment, we strongly recommend
a six-node cluster, with three masters and three replicas.

You can test this locally by creating the following directories named
after the port number of the instance you'll run inside any given directory.

For example:

[code example]

Create a `redis.conf` file inside each of the directories, from 7000 to 7005.
As a template for your configuration file just use the small example above,
but make sure to replace the port number `7000` with the right port number
according to the directory name.

You can start each instance as follows, each running in a separate terminal tab:

[code example]
You'll see from the logs that every node assigns itself a new ID:

    [82462] 26 Nov 11:56:55.329 * No cluster configuration found, I'm 97a3a64667477371c4479320d683e4c8db5858b1

This ID will be used forever by this specific instance in order for the instance
to have a unique name in the context of the cluster. Every node
remembers every other node using this IDs, and not by IP or port.
IP addresses and ports may change, but the unique node identifier will never
change for all the life of the node. We call this identifier simply **Node ID**.

#### Create a Redis Cluster

Now that we have a number of instances running, you need to create your cluster by writing some meaningful configuration to the nodes.

You can configure and execute individual instances manually or use the create-cluster script.
Let's go over how you do it manually.

To create the cluster, run:

    redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
    127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
    --cluster-replicas 1

The command used here is **create**, since we want to create a new cluster.
The option `--cluster-replicas 1` means that we want a replica for every master created.

The other arguments are the list of addresses of the instances I want to use
to create the new cluster.

`redis-cli` will propose a configuration. Accept the proposed configuration by typing **yes**.
The cluster will be configured and *joined*, which means that instances will be
bootstrapped into talking with each other. Finally, if everything has gone well, you'll see a message like this:

    [OK] All 16384 slots covered

This means that there is at least one master instance serving each of the
16384 available slots.

If you don't want to create a Redis Cluster by configuring and executing
individual instances manually as explained above, there is a much simpler
system (but you'll not learn the same amount of operational details).

Find the `utils/create-cluster` directory in the Redis distribution.
There is a script called `create-cluster` inside (same name as the directory
it is contained into), it's a simple bash script. In order to start
a 6 nodes cluster with 3 masters and 3 replicas just type the following
commands:

1. `create-cluster start`
2. `create-cluster create`

Reply to `yes` in step 2 when the `redis-cli` utility wants you to accept
the cluster layout.

You can now interact with the cluster, the first node will start at port 30001
by default. When you are done, stop the cluster with:

3. `create-cluster stop`

Please read the `README` inside this directory for more information on how
to run the script.

#### Interact with the cluster

To connect to Redis Cluster, you'll need a cluster-aware Redis client.
See the [documentation]() for your client of choice to determine its cluster support.

You can also test your Redis Cluster using the `redis-cli` command line utility:

[code example]

If you created the cluster using the script, your nodes may listen
on different ports, starting from 30001 by default.

The `redis-cli` cluster support is very basic, so it always uses the fact that
Redis Cluster nodes are able to redirect a client to the right node.
A serious client is able to do better than that, and cache the map between
hash slots and nodes addresses, to directly use the right connection to the
right node. The map is refreshed only when something changed in the cluster
configuration, for example after a failover or after the system administrator
changed the cluster layout by adding or removing nodes.

#### Write an example app with redis-rb-cluster

Before going forward showing how to operate the Redis Cluster, doing things
like a failover, or a resharding, we need to create some example application
or at least to be able to understand the semantics of a simple Redis Cluster
client interaction.

In this way we can run an example and at the same time try to make nodes
failing, or start a resharding, to see how Redis Cluster behaves under real
world conditions. It is not very helpful to see what happens while nobody
is writing to the cluster.

This section explains some basic usage of
[redis-rb-cluster](https://github.com/antirez/redis-rb-cluster) showing two
examples.
The first is the following, and is the
[`example.rb`](https://github.com/antirez/redis-rb-cluster/blob/master/example.rb)
file inside the redis-rb-cluster distribution:

[code example]

The application does a very simple thing, it sets keys in the form `foo<number>` to `number`, one after the other. So if you run the program the result is the
following stream of commands:

* SET foo0 0
* SET foo1 1
* SET foo2 2
* And so forth...

The program looks more complex than it should usually as it is designed to
show errors on the screen instead of exiting with an exception, so every
operation performed with the cluster is wrapped by `begin` `rescue` blocks.

The **line 14** is the first interesting line in the program. It creates the
Redis Cluster object, using as argument a list of *startup nodes*, the maximum
number of connections this object is allowed to take against different nodes,
and finally the timeout after a given operation is considered to be failed.

The startup nodes don't need to be all the nodes of the cluster. The important
thing is that at least one node is reachable. Also note that redis-rb-cluster
updates this list of startup nodes as soon as it is able to connect with the
first node. You should expect such a behavior with any other serious client.

Now that we have the Redis Cluster object instance stored in the **rc** variable,
we are ready to use the object like if it was a normal Redis object instance.

This is exactly what happens in **line 18 to 26**: when we restart the example
we don't want to start again with `foo0`, so we store the counter inside
Redis itself. The code above is designed to read this counter, or if the
counter does not exist, to assign it the value of zero.

However note how it is a while loop, as we want to try again and again even
if the cluster is down and is returning errors. Normal applications don't need
to be so careful.

**Lines between 28 and 37** start the main loop where the keys are set or
an error is displayed.

Note the `sleep` call at the end of the loop. In your tests you can remove
the sleep if you want to write to the cluster as fast as possible (relatively
to the fact that this is a busy loop without real parallelism of course, so
you'll get the usually 10k ops/second in the best of the conditions).

Normally writes are slowed down in order for the example application to be
easier to follow by humans.

Starting the application produces the following output:

[code example]

This is not a very interesting program and we'll use a better one in a moment
but we can already see what happens during a resharding when the program
is running.

#### Reshard the cluster

Now we are ready to try a cluster resharding. To do this, please
keep the example.rb program running, so that you can see if there is some
impact on the program running. Also, you may want to comment the `sleep`
call to have some more serious write load during resharding.

Resharding basically means to move hash slots from a set of nodes to another
set of nodes.
Like cluster creation, it is accomplished using the redis-cli utility.

To start a resharding, just type:

    redis-cli --cluster reshard 127.0.0.1:7000

You only need to specify a single node, redis-cli will find the other nodes
automatically.

Currently redis-cli is only able to reshard with the administrator support,
you can't just say move 5% of slots from this node to the other one (but
this is pretty trivial to implement). So it starts with questions. The first
is how much of a resharding do you want to do:

    How many slots do you want to move (from 1 to 16384)?

We can try to reshard 1000 hash slots, that should already contain a non
trivial amount of keys if the example is still running without the sleep
call.

Then redis-cli needs to know what is the target of the resharding, that is,
the node that will receive the hash slots.
I'll use the first master node, that is, 127.0.0.1:7000, but I need
to specify the Node ID of the instance. This was already printed in a
list by redis-cli, but I can always find the ID of a node with the following
command if I need:

[code example]

Ok so my target node is 97a3a64667477371c4479320d683e4c8db5858b1.

Now you'll get asked from what nodes you want to take those keys.
I'll just type `all` in order to take a bit of hash slots from all the
other master nodes.

After the final confirmation you'll see a message for every slot that
redis-cli is going to move from a node to another, and a dot will be printed
for every actual key moved from one side to the other.

While the resharding is in progress you should be able to see your
example program running unaffected. You can stop and restart it multiple times
during the resharding if you want.

At the end of the resharding, you can test the health of the cluster with
the following command:

    redis-cli --cluster check 127.0.0.1:7000

All the slots will be covered as usual, but this time the master at
127.0.0.1:7000 will have more hash slots, something around 6461.

Resharding can be performed automatically without the need to manually
enter the parameters in an interactive way. This is possible using a command
line like the following:

    redis-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes

This allows to build some automatism if you are likely to reshard often,
however currently there is no way for `redis-cli` to automatically
rebalance the cluster checking the distribution of keys across the cluster
nodes and intelligently moving slots as needed. This feature will be added
in the future.

The `--cluster-yes` option instructs the cluster manager to automatically answer
"yes" to the command's prompts, allowing it to run in a non-interactive mode.
Note that this option can also be activated by setting the
`REDISCLI_CLUSTER_YES` environment variable.

#### A more interesting example application

The example application we wrote early is not very good.
It writes to the cluster in a simple way without even checking if what was
written is the right thing.

From our point of view the cluster receiving the writes could just always
write the key `foo` to `42` to every operation, and we would not notice at
all.

So in the `redis-rb-cluster` repository, there is a more interesting application
that is called `consistency-test.rb`. It uses a set of counters, by default 1000, and sends [`INCR`](/commands/incr) commands in order to increment the counters.

However instead of just writing, the application does two additional things:

* When a counter is updated using [`INCR`](/commands/incr), the application remembers the write.
* It also reads a random counter before every write, and check if the value is what we expected it to be, comparing it with the value it has in memory.

What this means is that this application is a simple **consistency checker**,
and is able to tell you if the cluster lost some write, or if it accepted
a write that we did not receive acknowledgment for. In the first case we'll
see a counter having a value that is smaller than the one we remember, while
in the second case the value will be greater.

Running the consistency-test application produces a line of output every
second:

[code example]

The line shows the number of **R**eads and **W**rites performed, and the
number of errors (query not accepted because of errors since the system was
not available).

If some inconsistency is found, new lines are added to the output.
This is what happens, for example, if I reset a counter manually while
the program is running:

[code example]

When I set the counter to 0 the real value was 114, so the program reports
114 lost writes ([`INCR`](/commands/incr) commands that are not remembered by the cluster).

This program is much more interesting as a test case, so we'll use it
to test the Redis Cluster failover.

#### Test the failover

To trigger the failover, the simplest thing we can do (that is also
the semantically simplest failure that can occur in a distributed system)
is to crash a single process, in our case a single master.

During this test, you should take a tab open with the consistency test
application running.

We can identify a master and crash it with the following command:

[code example]

Ok, so 7000, 7001, and 7002 are masters. Let's crash node 7002 with the
**DEBUG SEGFAULT** command:

[code example]

Now we can look at the output of the consistency test to see what it reported.

[code example]

As you can see during the failover the system was not able to accept 578 reads and 577 writes, however no inconsistency was created in the database. This may
sound unexpected as in the first part of this tutorial we stated that Redis
Cluster can lose writes during the failover because it uses asynchronous
replication. What we did not say is that this is not very likely to happen
because Redis sends the reply to the client, and the commands to replicate
to the replicas, about at the same time, so there is a very small window to
lose data. However the fact that it is hard to trigger does not mean that it
is impossible, so this does not change the consistency guarantees provided
by Redis cluster.

We can now check what is the cluster setup after the failover (note that
in the meantime I restarted the crashed instance so that it rejoins the
cluster as a replica):

[code example]

Now the masters are running on ports 7000, 7001 and 7005. What was previously
a master, that is the Redis instance running on port 7002, is now a replica of
7005.

The output of the [`CLUSTER NODES`](/commands/cluster-nodes) command may look intimidating, but it is actually pretty simple, and is composed of the following tokens:

* Node ID
* ip:port
* flags: master, replica, myself, fail, ...
* if it is a replica, the Node ID of the master
* Time of the last pending PING still waiting for a reply.
* Time of the last PONG received.
* Configuration epoch for this node (see the Cluster specification).
* Status of the link to this node.
* Slots served...

#### Manual failover

Sometimes it is useful to force a failover without actually causing any problem
on a master. For example, to upgrade the Redis process of one of the
master nodes it is a good idea to failover it to turn it into a replica
with minimal impact on availability.

Manual failovers are supported by Redis Cluster using the [`CLUSTER FAILOVER`](/commands/cluster-failover)
command, that must be executed in one of the replicas of the master you want
to failover.

Manual failovers are special and are safer compared to failovers resulting from
actual master failures. They occur in a way that avoids data loss in the
process, by switching clients from the original master to the new master only
when the system is sure that the new master processed all the replication stream
from the old one.

This is what you see in the replica log when you perform a manual failover:

    # Manual failover user request accepted.
    # Received replication offset for paused master manual failover: 347540
    # All master replication stream processed, manual failover can start.
    # Start of election delayed for 0 milliseconds (rank #0, offset 347540).
    # Starting a failover election for epoch 7545.
    # Failover election won: I'm the new master.

Basically clients connected to the master we are failing over are stopped.
At the same time the master sends its replication offset to the replica, that
waits to reach the offset on its side. When the replication offset is reached,
the failover starts, and the old master is informed about the configuration
switch. When the clients are unblocked on the old master, they are redirected
to the new master.

To promote a replica to master, it must first be known as a replica by a majority of the masters in the cluster.
  Otherwise, it cannot win the failover election.
  If the replica has just been added to the cluster (see [Add a new node as a replica](#add-a-new-node-as-a-replica)), you may need to wait a while before sending the [`CLUSTER FAILOVER`](/commands/cluster-failover) command, to make sure the masters in cluster are aware of the new replica.

#### Add a new node

Adding a new node is basically the process of adding an empty node and then
moving some data into it, in case it is a new master, or telling it to
setup as a replica of a known node, in case it is a replica.

We'll show both, starting with the addition of a new master instance.

In both cases the first step to perform is **adding an empty node**.

This is as simple as to start a new node in port 7006 (we already used
from 7000 to 7005 for our existing 6 nodes) with the same configuration
used for the other nodes, except for the port number, so what you should
do in order to conform with the setup we used for the previous nodes:

* Create a new tab in your terminal application.
* Enter the `cluster-test` directory.
* Create a directory named `7006`.
* Create a redis.conf file inside, similar to the one used for the other nodes but using 7006 as port number.
* Finally start the server with `../redis-server ./redis.conf`

At this point the server should be running.

Now we can use **redis-cli** as usual in order to add the node to
the existing cluster.

    redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

As you can see I used the **add-node** command specifying the address of the
new node as first argument, and the address of a random existing node in the
cluster as second argument.

In practical terms redis-cli here did very little to help us, it just
sent a [`CLUSTER MEET`](/commands/cluster-meet) message to the node, something that is also possible
to accomplish manually. However redis-cli also checks the state of the
cluster before to operate, so it is a good idea to perform cluster operations
always via redis-cli even when you know how the internals work.

Now we can connect to the new node to see if it really joined the cluster:

[code example]

Note that since this node is already connected to the cluster it is already
able to redirect client queries correctly and is generally speaking part of
the cluster. However it has two peculiarities compared to the other masters:

* It holds no data as it has no assigned hash slots.
* Because it is a master without assigned slots, it does not participate in the election process when a replica wants to become a master.

Now it is possible to assign hash slots to this node using the resharding
feature of `redis-cli`.
It is basically useless to show this as we already
did in a previous section, there is no difference, it is just a resharding
having as a target the empty node.

##### Add a new node as a replica

Adding a new replica can be performed in two ways. The obvious one is to
use redis-cli again, but with the --cluster-slave option, like this:

    redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-slave

Note that the command line here is exactly like the one we used to add
a new master, so we are not specifying to which master we want to add
the replica. In this case, what happens is that redis-cli will add the new
node as replica of a random master among the masters with fewer replicas.

However you can specify exactly what master you want to target with your
new replica with the following command line:

    redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-slave --cluster-master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

This way we assign the new replica to a specific master.

A more manual way to add a replica to a specific master is to add the new
node as an empty master, and then turn it into a replica using the
[`CLUSTER REPLICATE`](/commands/cluster-replicate) command. This also works if the node was added as a replica
but you want to move it as a replica of a different master.

For example in order to add a replica for the node 127.0.0.1:7005 that is
currently serving hash slots in the range 11423-16383, that has a Node ID
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e, all I need to do is to connect
with the new node (already added as empty master) and send the command:

    redis 127.0.0.1:7006> cluster replicate 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

That's it. Now we have a new replica for this set of hash slots, and all
the other nodes in the cluster already know (after a few seconds needed to
update their config). We can verify with the following command:

[code example]

The node 3c3a0c... now has two replicas, running on ports 7002 (the existing one) and 7006 (the new one).

#### Remove a node

To remove a replica node just use the `del-node` command of redis-cli:

    redis-cli --cluster del-node 127.0.0.1:7000 `<node-id>`

The first argument is just a random node in the cluster, the second argument
is the ID of the node you want to remove.

You can remove a master node in the same way as well, **however in order to
remove a master node it must be empty**. If the master is not empty you need
to reshard data away from it to all the other master nodes before.

An alternative to remove a master node is to perform a manual failover of it
over one of its replicas and remove the node after it turned into a replica of the
new master. Obviously this does not help when you want to reduce the actual
number of masters in your cluster, in that case, a resharding is needed.

There is a special scenario where you want to remove a failed node.
You should not use the `del-node` command because it tries to connect to all nodes and you will encounter a "connection refused" error.
Instead, you can use the `call` command:

    redis-cli --cluster call 127.0.0.1:7000 cluster forget `<node-id>`

This command will execute [`CLUSTER FORGET`](/commands/cluster-forget) command on every node.

#### Replica migration

In Redis Cluster, you can reconfigure a replica to replicate with a
different master at any time just using this command:

    CLUSTER REPLICATE <master-node-id>

However there is a special scenario where you want replicas to move from one
master to another one automatically, without the help of the system administrator.
The automatic reconfiguration of replicas is called *replicas migration* and is
able to improve the reliability of a Redis Cluster.

You can read the details of replicas migration in the [Redis Cluster Specification](), here we'll only provide some information about the
general idea and what you should do in order to benefit from it.

The reason why you may want to let your cluster replicas to move from one master
to another under certain condition, is that usually the Redis Cluster is as
resistant to failures as the number of replicas attached to a given master.

For example a cluster where every master has a single replica can't continue
operations if the master and its replica fail at the same time, simply because
there is no other instance to have a copy of the hash slots the master was
serving. However while net-splits are likely to isolate a number of nodes
at the same time, many other kind of failures, like hardware or software failures
local to a single node, are a very notable class of failures that are unlikely
to happen at the same time, so it is possible that in your cluster where
every master has a replica, the replica is killed at 4am, and the master is killed
at 6am. This still will result in a cluster that can no longer operate.

To improve reliability of the system we have the option to add additional
replicas to every master, but this is expensive. Replica migration allows to
add more replicas to just a few masters. So you have 10 masters with 1 replica
each, for a total of 20 instances. However you add, for example, 3 instances
more as replicas of some of your masters, so certain masters will have more
than a single replica.

With replicas migration what happens is that if a master is left without
replicas, a replica from a master that has multiple replicas will migrate to
the *orphaned* master. So after your replica goes down at 4am as in the example
we made above, another replica will take its place, and when the master
will fail as well at 5am, there is still a replica that can be elected so that
the cluster can continue to operate.

So what you should know about replicas migration in short?

* The cluster will try to migrate a replica from the master that has the greatest number of replicas in a given moment.
* To benefit from replica migration you have just to add a few more replicas to a single master in your cluster, it does not matter what master.
* There is a configuration parameter that controls the replica migration feature that is called `cluster-migration-barrier`: you can read more about it in the example `redis.conf` file provided with Redis Cluster.

#### Upgrade nodes in a Redis Cluster

Upgrading replica nodes is easy since you just need to stop the node and restart
it with an updated version of Redis. If there are clients scaling reads using
replica nodes, they should be able to reconnect to a different replica if a given
one is not available.

Upgrading masters is a bit more complex, and the suggested procedure is:

1. Use [`CLUSTER FAILOVER`](/commands/cluster-failover) to trigger a manual failover of the master to one of its replicas.
   (See the [Manual failover](#manual-failover) in this topic.)
2. Wait for the master to turn into a replica.
3. Finally upgrade the node as you do for replicas.
4. If you want the master to be the node you just upgraded, trigger a new manual failover in order to turn back the upgraded node into a master.

Following this procedure you should upgrade one node after the other until
all the nodes are upgraded.

#### Migrate to Redis Cluster

Users willing to migrate to Redis Cluster may have just a single master, or
may already using a preexisting sharding setup, where keys
are split among N nodes, using some in-house algorithm or a sharding algorithm
implemented by their client library or Redis proxy.

In both cases it is possible to migrate to Redis Cluster easily, however
what is the most important detail is if multiple-keys operations are used
by the application, and how. There are three different cases:

1. Multiple keys operations, or transactions, or Lua scripts involving multiple keys, are not used. Keys are accessed independently (even if accessed via transactions or Lua scripts grouping multiple commands, about the same key, together).
2. Multiple keys operations, or transactions, or Lua scripts involving multiple keys are used but only with keys having the same **hash tag**, which means that the keys used together all have a `{...}` sub-string that happens to be identical. For example the following multiple keys operation is defined in the context of the same hash tag: `SUNION {user:1000}.foo {user:1000}.bar`.
3. Multiple keys operations, or transactions, or Lua scripts involving multiple keys are used with key names not having an explicit, or the same, hash tag.

The third case is not handled by Redis Cluster: the application requires to
be modified in order to not use multi keys operations or only use them in
the context of the same hash tag.

Case 1 and 2 are covered, so we'll focus on those two cases, that are handled
in the same way, so no distinction will be made in the documentation.

Assuming you have your preexisting data set split into N masters, where
N=1 if you have no preexisting sharding, the following steps are needed
in order to migrate your data set to Redis Cluster:

1. Stop your clients. No automatic live-migration to Redis Cluster is currently possible. You may be able to do it orchestrating a live migration in the context of your application / environment.
2. Generate an append only file for all of your N masters using the [`BGREWRITEAOF`](/commands/bgrewriteaof) command, and waiting for the AOF file to be completely generated.
3. Save your AOF files from aof-1 to aof-N somewhere. At this point you can stop your old instances if you wish (this is useful since in non-virtualized deployments you often need to reuse the same computers).
4. Create a Redis Cluster composed of N masters and zero replicas. You'll add replicas later. Make sure all your nodes are using the append only file for persistence.
5. Stop all the cluster nodes, substitute their append only file with your pre-existing append only files, aof-1 for the first node, aof-2 for the second node, up to aof-N.
6. Restart your Redis Cluster nodes with the new AOF files. They'll complain that there are keys that should not be there according to their configuration.
7. Use `redis-cli --cluster fix` command in order to fix the cluster so that keys will be migrated according to the hash slots each node is authoritative or not.
8. Use `redis-cli --cluster check` at the end to make sure your cluster is ok.
9. Restart your clients modified to use a Redis Cluster aware client library.

There is an alternative way to import data from external instances to a Redis
Cluster, which is to use the `redis-cli --cluster import` command.

The command moves all the keys of a running instance (deleting the keys from
the source instance) to the specified pre-existing Redis Cluster. However
note that if you use a Redis 2.8 instance as source instance the operation
may be slow since 2.8 does not implement migrate connection caching, so you
may want to restart your source instance with a Redis 3.x version before
to perform such operation.

Starting with Redis 5, if not for backward compatibility, the Redis project no longer uses the word slave. Unfortunately in this command the word slave is part of the protocol, so we'll be able to remove such occurrences only when this API will be naturally deprecated.

Learn more

* [Redis Cluster specification]()
* [Linear Scaling with Redis Enterprise](https://redis.com/redis-enterprise/technology/linear-scaling-redis-enterprise/)
* [Docker documentation](https://docs.docker.com/engine/userguide/networking/dockernetworks/)

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

mkdir cluster-test
cd cluster-test
mkdir 7000 7001 7002 7003 7004 7005

$ redis-cli -c -p 7000
redis 127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
redis 127.0.0.1:7002> set hello world
-> Redirected to slot [866] located at 127.0.0.1:7000
OK
redis 127.0.0.1:7000> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
redis 127.0.0.1:7002> get hello
-> Redirected to slot [866] located at 127.0.0.1:7000
"world"

1  require './cluster'
   2
   3  if ARGV.length != 2
   4      startup_nodes = [
   5          {:host => "127.0.0.1", :port => 7000},
   6          {:host => "127.0.0.1", :port => 7001}
   7      ]
   8  else
   9      startup_nodes = [
  10          {:host => ARGV[0], :port => ARGV[1].to_i}
  11      ]
  12  end
  13
  14  rc = RedisCluster.new(startup_nodes,32,:timeout => 0.1)
  15
  16  last = false
  17
  18  while not last
  19      begin
  20          last = rc.get("__last__")
  21          last = 0 if !last
  22      rescue => e
  23          puts "error #{e.to_s}"
  24          sleep 1
  25      end
  26  end
  27
  28  ((last.to_i+1)..1000000000).each{|x|
  29      begin
  30          rc.set("foo#{x}",x)
  31          puts rc.get("foo#{x}")
  32          rc.set("__last__",x)
  33      rescue => e
  34          puts "error #{e.to_s}"
  35      end
  36      sleep 0.1
  37  }

ruby ./example.rb
1
2
3
4
5
6
7
8
9
^C (I stopped the program here)

$ redis-cli -p 7000 cluster nodes | grep myself
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5460

$ redis-cli -p 7002 debug segfault
Error: Server closed the connection