How do I simulate a Redis Enteprise Cluster failure?

Question

How do I simulate a Redis Enterprise Cluster failure?

Answer

A Redis Enterprise cluster failure can be simulated by using the firewall rules to block traffic to the database endpoint port. If that is not possible, you can perform the following (in each of the Redis cluster nodes):

Stop the node watchdog process  supervisorctl stop node_wd Then stop the proxy/endpoint process, so that connections to all the database(s) would fail, thus leading to the LB health checks failing.  supervisorctl stop dmcproxy And to resume  supervisorctl start node_wd

A good script for simulating network cable disconnection is as below.

#!/bin/bash

# usage: ./faildrop.sh 60
# will default to 90 seconds otherwise
#

s1=${1:-90}

iptables -t filter -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
iptables -t filter -A INPUT -j DROP
iptables -t filter -A OUTPUT -p tcp -m tcp --sport 22 -j ACCEPT
iptables -t filter -A OUTPUT -j DROP

echo Killed, waiting ${s1} seconds...
sleep ${s1}

echo Resuming...

iptables -F INPUT
iptables -F OUTPUT

echo Done.