Learn about Redis Cluster architecture, setup, and operations. Master horizontal scaling, sharding, and high availability in Redis deployments.
Learn about Redis Cluster architecture, setup, and operations. Master horizontal scaling, sharding, and high availability in Redis deployments.
Redis Cluster is the official distributed implementation of Redis, providing horizontal scaling
through automatic data sharding across multiple Redis nodes. Unlike master-replica replication,
which primarily addresses high availability and read scaling, Redis Cluster enables both data sharding
and high availability, allowing Redis deployments to grow beyond the memory limits of a single server.
In this lesson, we'll explore the architecture and operation of Redis Cluster, including setup,
configuration, sharding mechanics, and high availability features. We'll also revisit Sentinel in more
detail, comparing its approach to high availability with that of Redis Cluster.
Redis Cluster was introduced in Redis 3.0 and has become the preferred method for scaling Redis
horizontally in production environments. It provides automatic data sharding across nodes while also
managing replica nodes for each shard to ensure high availability.
Redis Cluster implements a peer-to-peer architecture where each node is connected to every other node
through a gossip protocol. There is no central coordinator; instead, nodes communicate with each other
to maintain a consistent view of the cluster state.
Master Nodes
: Each master node is responsible for a subset of the keyspace,
determined by the cluster's sharding mechanism.
Replica Nodes
: Each master can have one or more replica nodes for redundancy.
If a master fails, one of its replicas can be promoted to master.
Hash Slots
: The keyspace is divided into 16,384 hash slots. Each master node
is assigned a subset of these slots, and keys are mapped to slots using a hash function.
Gossip Protocol
: Nodes exchange information about the cluster state, including node
health, slot assignments, and migration status.
Cluster Bus
: A separate TCP port (default: node port + 10000) used by nodes to
communicate with each other.
Redis Cluster Architecture Diagram
The key feature of Redis Cluster is its ability to distribute data across multiple nodes using a sharding
mechanism based on hash slots:
16,384 Hash Slots
: The Redis keyspace is divided into these slots, numbered 0 to 16,383.
Key Distribution
: Each key is mapped to a hash slot using the CRC16 algorithm modulo 16384:
HASH_SLOT = CRC16(key) mod 16384
Slot Assignment
: Each master node is responsible for a subset of these hash slots.
For example, in a 3-node cluster, Master A might handle slots 0-5460, Master B slots 5461-10922, and
Master C slots 10923-16383.
: Keys can be forced to the same hash slot by enclosing a part of the key
in curly braces. The hash is calculated only on the part inside the braces:
user: :profile and user: :sessions
These keys will be assigned to the same hash slot because the hashing only considers "1000".
Redis Cluster provides built-in high availability through:
Master-Replica Model
: Each master node can have one or more replica nodes
that maintain a copy of the master's data.
Automatic Failover
: If a master node fails, the cluster will automatically
promote one of its replicas to become the new master.
Split Brain Prevention
: Redis Cluster requires a majority of master nodes to
be reachable for automatic failover to occur, preventing the "split-brain" scenario.
Node Failure Detection
: Nodes use heartbeat messages to detect the failure
of other nodes. A node is considered down if it hasn't responded to pings for a certain period.
Let's walk through the process of setting up a basic Redis Cluster:
For each Redis node, create a
file with cluster settings:
Repeat this for each node (7001, 7002, etc.), adjusting the port numbers and file paths.
1
2# Basic Redis configuration
3port 7000 # Port for this node (increment for each node)
4daemonize yes # Run as daemon
5pidfile /var/run/redis-7000.pid # PID file
6logfile /var/log/redis-7000.log # Log file
7dir /var/lib/redis/7000 # Data directory
8
9# Cluster specific configuration
10cluster-enabled yes # Enable cluster mode
11cluster-config-file nodes-7000.conf # Cluster node config file
12cluster-node-timeout 5000 # Node timeout in milliseconds
13 Start each Redis instance using its configuration file:
1
2# Start Redis instances
3redis-server /path/to/7000/redis.conf
4redis-server /path/to/7001/redis.conf
5redis-server /path/to/7002/redis.conf
6redis-server /path/to/7003/redis.conf
7redis-server /path/to/7004/redis.conf
8redis-server /path/to/7005/redis.conf
9
10# Verify that instances are running
11ps aux | grep redis-server
12 utility to create the cluster:
--cluster-replicas 1
option specifies that each master should have one replica.
In this case, the first three nodes (7000, 7001, 7002) will be masters, and the other three
(7003, 7004, 7005) will be replicas.
will display a plan showing which nodes will be masters and which will be
replicas, and ask for confirmation before creating the cluster:
Type 'yes' to confirm. The tool will proceed to create the cluster by connecting the nodes and
allocating hash slots.
1
2# Create cluster with 3 masters and 3 replicas
3redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 \\
4 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \\
5 --cluster-replicas 1
6 1
2>>> Performing hash slots allocation on 6 nodes...
3Master[0] -> Slots 0 - 5460
4Master[1] -> Slots 5461 - 10922
5Master[2] -> Slots 10923 - 16383
6Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
7Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
8Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
9>>> Trying to optimize slaves allocation for anti-affinity
10[OK] Perfect anti-affinity obtained!
11M: 67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1 127.0.0.1:7000
12 slots:[0-5460] (5461 slots) master
13M: 292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f 127.0.0.1:7001
14 slots:[5461-10922] (5462 slots) master
15M: 6ec23923021cf3ffec47632106199cb7f496ce01 127.0.0.1:7002
16 slots:[10923-16383] (5461 slots) master
17S: 824fe116063c5e9fdd1d2591e8dae97538dfe0fd 127.0.0.1:7003
18 replicates 67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1
19S: 31259a5cb676aa9302142a0b6e8cf1e7eadca8cd 127.0.0.1:7004
20 replicates 292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f
21S: 28de711476e9ee352db4377ec7d4a2877e8e06d1 127.0.0.1:7005
22 replicates 6ec23923021cf3ffec47632106199cb7f496ce01
23Can I set the above configuration? (type 'yes' to accept):
24 Check the cluster status to verify that it's working correctly:
When using
flag (cluster mode), if you try to access a
key that doesn't belong to the node you're connected to, the client will automatically follow redirects
to the correct node.
1
2# Connect to any node in cluster mode
3redis-cli -c -p 7000
4
5# Check cluster status
6CLUSTER INFO
7
8# List cluster nodes
9CLUSTER NODES
10
11# Test cluster operations
12SET mykey "Hello Cluster"
13GET mykey
14 Learn about Redis Cluster Operations.
To add a new node to an existing cluster:
Start a new Redis instance in cluster mode:
Add the node to the cluster as a master (without slots initially):
The second address (7000) is any existing node in the cluster.
Alternatively, add the node as a replica of an existing master:
master-node-id
is the ID of the master node, which can be found using
CLUSTER NODES
1
2# Start a new Redis instance
3redis-server /path/to/7006/redis.conf
4 1
2# Add node to cluster
3redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000
4 1
2# Add node as replica
3redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 \\
4 --cluster-slave --cluster-master-id <master-node-id>
5 To redistribute hash slots among nodes (e.g., after adding a new master node):
The tool will prompt you for:
The number of slots to move (e.g., 4000)
The ID of the node receiving the slots
The source nodes to take slots from (specific nodes or 'all')
1
2# Reshard the cluster
3redis-cli --cluster reshard 127.0.0.1:7000
4 To remove a node from the cluster:
If it's a master node with slots, first reshard its slots to other nodes:
When prompted, move all slots from the target node to other nodes.
Remove the node from the cluster:
is the ID of the node to remove, which can be found using
CLUSTER NODES
1
2# Reshard all slots away from the node
3redis-cli --cluster reshard 127.0.0.1:7000
4 1
2# Remove node
3redis-cli --cluster del-node 127.0.0.1:7000 <node-id>
4 To evenly distribute hash slots across all master nodes:
This command analyses the current distribution of hash slots and moves slots to achieve a balanced distribution.
1
2# Rebalance the cluster
3redis-cli --cluster rebalance 127.0.0.1:7000
4 Here are some useful commands for managing and monitoring a Redis Cluster:
Description
CLUSTER INFO
Display information about the cluster state
CLUSTER NODES
List all nodes in the cluster with their status
CLUSTER SLOTS
Show slots range assigned to each node
CLUSTER KEYSLOT key
Calculate which hash slot a key belongs to
CLUSTER COUNTKEYSINSLOT slot
Count keys in a specific hash slot
CLUSTER MEET ip port
Force a node to meet another node
CLUSTER FAILOVER
Force a replica to failover to master
CLUSTER FORGET nodeid
Remove a node from a node's view of the cluster
CLUSTER REPLICATE nodeid
Configure a node to replicate another node
Learn about Failover and High Availability in Redis Cluster.
Redis Cluster provides automatic failover when a master node becomes unavailable:
Node Failure Detection
: Nodes exchange PING messages to detect the health
of other nodes. If a node doesn't respond for a certain period (defined by
cluster-node-timeout
),
it's flagged as potentially down.
: A majority of master nodes must agree that a master is down for failover to begin.
This prevents failovers in network partition scenarios where a minority of nodes are separated from the majority.
Replica Election
: The replicas of the failed master self-elect a candidate to be promoted to master.
Typically, the replica with the most up-to-date data is chosen.
Configuration Update
: The new master updates the cluster configuration and takes ownership of
the hash slots that belonged to the failed master.
Cluster Convergence
: The configuration change is propagated to all nodes, and clients are
informed of the new slot mapping.
Redis Cluster includes safeguards against "split-brain" scenarios, where network partitions might
result in multiple masters for the same hash slots:
Majority Rule
: Automatic failover only occurs if the majority of master nodes
can communicate with each other. This ensures that in a network partition, only the partition
with the majority of masters can reconfigure the cluster.
Partition Tolerance
: The minority partition will stop accepting write operations
once it detects that it cannot reach a majority of master nodes. This prevents data inconsistency
between partitions.
Redis Cluster provides "availability over consistency" in certain edge cases. If a minority partition continues
to accept reads for existing data (even though it can't accept writes), clients reading from this partition
might see stale data compared to the majority partition.
You can also trigger a manual failover, which is useful for planned maintenance:
Connect to a replica node:
Trigger failover from the replica:
During a manual failover, the replica coordinates with its master to ensure a clean transition with no data loss.
The process works as follows:
The replica contacts its master to stop processing client commands.
The master sends the replica its latest replication offset.
The replica waits until it has processed all data up to that offset.
The replica promotes itself to master and updates the cluster configuration.
The old master demotes itself to replica and starts replicating from the new master.
There are also variations of the
CLUSTER FAILOVER
command for different scenarios:
CLUSTER FAILOVER FORCE
: Skip coordination with the master (use only when the master is unreachable).
CLUSTER FAILOVER TAKEOVER
: Force promotion even without majority agreement (emergency use only).
1
2# Connect to replica
3redis-cli -c -p 7003
4 1
2# Manual failover
3CLUSTER FAILOVER
4 Clients need to be "cluster-aware" to work with Redis Cluster effectively:
Redis Cluster clients need to handle several cluster-specific behaviors:
Redirection Handling
: When a client sends a command for a key that doesn't belong to
the contacted node, the client receives a
redirection error and must redirect
the command to the correct node.
Connection Pooling
: Clients typically maintain connections to multiple (or all)
nodes in the cluster.
Cluster State Caching
: Clients cache the mapping between hash slots and nodes
to minimize redirections.
Retry Logic
: Handling temporary failures and cluster reconfiguration events.
Here's an example of using a cluster-aware client library (Node.js with ioredis):
1
2const Redis = require('ioredis');
3
4// Create a Redis Cluster client
5const cluster = new Redis.Cluster([
6 { host: '127.0.0.1', port: 7000 },
7 { host: '127.0.0.1', port: 7001 },
8 { host: '127.0.0.1', port: 7002 }
9], {
10 // Enable read operations on replicas
11 scaleReads: 'slave',
12
13 // Retry on errors
14 retryDelayOnFailover: 100,
15 retryDelayOnClusterDown: 100,
16 retryDelayOnTryAgain: 100,
17 maxRetriesPerRequest: 3,
18
19 // Refresh cluster topology
20 enableReadyCheck: true,
21 enableOfflineQueue: true,
22 redisOptions: {
23 password: 'your-password',
24 connectTimeout: 10000,
25 lazyConnect: true,
26 }
27});
28
29// Basic operations work the same as regular Redis
30cluster.set('foo', 'bar');
31cluster.get('foo', (err, res) => {
32 console.log(res); // 'bar'
33});
34
35// Using hash tags to force keys into the same slot
36// This is necessary for multi-key operations
37cluster.mset('{user:1}:name', 'John', '{user:1}:email', '[email protected]');
38cluster.mget('{user:1}:name', '{user:1}:email', (err, res) => {
39 console.log(res); // ['John', '[email protected]']
40});
41
42// Close connections when done
43cluster.disconnect();
44 One important limitation of Redis Cluster is that multi-key operations are only supported when all keys belong
to the same hash slot. This is because Redis Cluster doesn't have built-in support for distributed transactions
across multiple nodes.
To work around this limitation, you can use hash tags to ensure related keys map to the same hash slot:
1
2# Check which hash slot a key belongs to
3CLUSTER KEYSLOT user:1000:profile # e.g., returns 9252
4CLUSTER KEYSLOT user:1000:sessions # Different slot, e.g., 3645
5
6# Using hash tags to force keys into the same slot
7CLUSTER KEYSLOT {user:1000}:profile # e.g., returns 7008
8CLUSTER KEYSLOT {user:1000}:sessions # Same slot, 7008
9
10# Now multi-key operations will work
11MSET {user:1000}:profile "John Doe" {user:1000}:sessions "3"
12MGET {user:1000}:profile {user:1000}:sessions
13 While we covered Redis Sentinel briefly in the previous lesson, let's explore it in more detail and
compare it with Redis Cluster:
Redis Sentinel is a distributed system designed to manage Redis instance failover:
Independent Processes
: Sentinel instances run as separate processes from Redis.
Monitoring
: Sentinels continuously monitor Redis master and replica instances.
Automatic Failover
: If a master is detected as down, Sentinels coordinate to
select and promote a replica to master.
Client Notifications
: Sentinels notify clients about master changes.
Configuration Provider
: Clients can query Sentinel to discover the current
master address.
Redis Sentinel Architecture Diagram
Redis Sentinel
Redis Cluster
Primary Focus
High Availability
High Availability + Sharding
Data Distribution
Each master holds the complete dataset
Data sharded across multiple masters
Horizontal Scaling
Read scaling only (via replicas)
Read and write scaling (via sharding)
Architecture
Separate monitoring processes
Integrated into Redis nodes
Master-replica
Multiple master-replica groups
Client Complexity
Lower (only master discovery)
Higher (slot mapping, redirects)
Setup Complexity
More complex
Multi-key Operations
Fully supported
Limited to same hash slot
Redis Sentinel
Your dataset fits entirely in a single Redis instance's memory
You need high availability but not horizontal scaling for writes
You rely heavily on multi-key operations
You prefer a simpler setup and client configuration
You want a separate monitoring system independent of Redis itself
Redis Cluster
Your dataset is larger than a single Redis instance's memory
You need to scale writes horizontally
You can design your application to work with key distribution and hash slots
You can group related keys using hash tags for multi-key operations
You need integrated scaling and high availability in a single system
Some organizations use both: Sentinel for smaller, critical datasets where multi-key operations
are important, and Cluster for large-scale data that benefits from sharding.
Learn about Best Practices for Redis Cluster.
Node Placement
: Deploy master nodes and their replicas on different physical
servers, racks, or availability zones for better fault tolerance.
Node Count
: Start with at least 3 master nodes and 3 replica nodes (one replica per master)
for a proper quorum and failover capability.
Network Configuration
: Ensure low-latency, reliable networking between nodes, as
cluster operations depend heavily on inter-node communication.
Resource Allocation
: Size nodes based on expected data volume, allowing enough
memory for data, Redis overhead, and operating system overhead.
Key Design
: Design keys with sharding in mind, particularly using hash tags to
ensure related keys are stored on the same node when needed.
Avoid Cross-Slot Operations
: Minimize operations that require keys from multiple
hash slots, as these won't work directly in Redis Cluster.
Client Retry Logic
: Implement proper retry logic in clients to handle temporary
cluster reconfiguration events.
Read Scaling
: Consider reading from replicas for read-heavy workloads, but be aware
of potential data inconsistency due to asynchronous replication.
Regular Monitoring
: Monitor cluster state, memory usage, and node health using
tools like
CLUSTER INFO
CLUSTER NODES
Planned Maintenance
: Use manual failover for planned maintenance to minimize disruption.
Backup Strategy
: Implement a backup strategy that works with a sharded architecture,
such as backing up each master individually or using Redis Enterprise features.
Scaling Plan
: Plan for cluster growth by regularly evaluating memory usage and
performance metrics, adding nodes before reaching capacity limits.
Rebalancing
: Periodically check slot distribution and rebalance if necessary to
maintain even data distribution.
Timeout Settings
cluster-node-timeout
based on your network
reliability. Higher values reduce false positives but increase failover time.
Memory Policies
: Configure appropriate
maxmemory-policy
settings for each node to prevent out-of-memory errors.
Persistence
: Configure persistence appropriate to your durability requirements,
balancing data safety with performance.
TCP Keepalive
tcp-keepalive
to detect stale connections,
particularly important in cloud environments.
Redis Cluster provides a powerful solution for horizontally scaling Redis while maintaining high availability.
By distributing data across multiple nodes through sharding, Redis Cluster can handle datasets much larger
than a single server's memory capacity, while also ensuring resilience through automatic failover.
Key takeaways from this lesson:
Redis Cluster shards data across nodes using a 16,384 hash slot mechanism
Each master node manages a subset of hash slots, with replica nodes for redundancy
Automatic failover ensures high availability when a master node fails
Cluster-aware clients handle redirections and maintain the cluster topology
Hash tags allow related keys to be stored on the same node for multi-key operations
Redis Sentinel provides an alternative high-availability solution without sharding
The choice between Redis Cluster and Redis Sentinel depends on your specific requirements for
scaling, high availability, and application design. For many large-scale applications, Redis Cluster
provides the ideal balance of horizontal scaling and resilience, making it a popular choice for
production deployments.
In the next lesson, we'll explore common Redis use cases and patterns, showing how Redis can be applied
to solve real-world problems efficiently.
Your progress is saved in this browser, on this device only.