Redis Persistence & Replication

While Redis is primarily an in-memory data store, it provides several mechanisms to persist data to disk. This persistence

is crucial for data durability, allowing Redis to recover its state after a restart or crash, and to back up data for

disaster recovery.

Redis offers two primary persistence mechanisms: RDB (Redis Database) and AOF (Append Only File). These can be used

independently or together to achieve different trade-offs between performance, data safety, and recovery speed.

Without persistence, all data in Redis would be lost when the server is restarted or crashes. Understanding persistence

options is essential for building reliable applications that use Redis for more than just ephemeral caching.

The RDB (Redis Database) persistence mechanism performs point-in-time snapshots of your Redis dataset at specified intervals.

Redis forks a child process to handle the snapshot creation while the parent process continues to serve requests.

The child process writes the entire dataset to a temporary file.

Once the temporary file is ready, it replaces the old RDB file (typically

RDB snapshots are configured in the Redis configuration file (

) using the

directive:

These settings can be adjusted according to your data volatility and durability needs. You can also trigger a snapshot manually:

1
2# Save the database to disk:
3#   save <seconds> <changes>
4save 900 1      # Save after 900 sec (15 min) if at least 1 change
5save 300 10     # Save after 300 sec (5 min) if at least 10 changes
6save 60 10000   # Save after 60 sec (1 min) if at least 10000 changes
7        

1
2# Synchronous save (blocks Redis until completed)
3SAVE
4
5# Asynchronous save (forks a child process)
6BGSAVE
7        

Compact file format

: RDB creates a single compact file, ideal for backups and disaster recovery.

Better performance

: RDB has minimal impact on Redis performance since it forks a separate process.

Faster restarts

: Redis can restart quickly by loading RDB files, which are optimized for reading.

Point-in-time snapshots

: Great for backup strategies requiring specific recovery points.

Data loss window

: Data changes since the last snapshot can be lost in a crash.

Fork process overhead

: On systems with large datasets, the fork operation can cause latency spikes.

Less frequent saves

: RDB typically runs at longer intervals, increasing the potential data loss window.

RDB should not be your only persistence option if you cannot afford to lose the most recent minutes of data in case of

a crash. For stronger durability guarantees, consider using AOF or a combination of both mechanisms.

The AOF (Append Only File) persistence logs every write operation received by the Redis server, which can then be replayed when

the server starts up, reconstructing the original dataset.

Redis logs every write command that modifies the dataset to an AOF file.

These commands are stored in the same format as the Redis protocol, making them readable and replicable.

When Redis restarts, it re-executes the commands from the AOF file to rebuild the dataset.

Redis can automatically rewrite the AOF file to optimize its size by replacing multiple operations on the same key

with a single operation reflecting the final state.

AOF is configured in the Redis configuration file using the following directives:

appendfsync

option controls how often Redis syncs the AOF to disk:

: Data safety is maximized but performance is impacted (sync after every write).

: Good compromise between safety and performance (sync once per second).

: Leaves sync to the operating system, best performance but higher risk of data loss.

You can also trigger an AOF rewrite manually to optimize the file size:

1
2# Enable AOF persistence
3appendonly yes
4
5# AOF filename
6appendfilename "appendonly.aof"
7
8# fsync policy: always, everysec, or no
9appendfsync everysec
10
11# Automatic AOF rewrite settings
12auto-aof-rewrite-percentage 100
13auto-aof-rewrite-min-size 64mb
14        

1
2# Asynchronously rewrite the AOF file
3BGREWRITEAOF
4        

Better durability

: AOF provides near-continuous persistence with minimal data loss.

Automatic rewrites

: Redis can compact the AOF file to prevent unbounded growth.

Human-readable format

: AOF files contain Redis commands, making them easier to inspect and repair if corrupted.

Safer defaults

appendfsync everysec

, you'll only lose at most one second of data.

Larger files

: AOF files are typically larger than RDB files for the same dataset.

Slower restarts

: Redis needs to replay all the commands in the AOF file, which can take longer.

Possible performance impact

appendfsync always

, every write operation requires a disk sync.

More complex to manage

: AOF files may require occasional manual analysis or repair.

Persistence Model

Point-in-time snapshots

Command log (write operations)

Data Safety

Can lose data since last snapshot

Minimal data loss (1 second with everysec)

Performance Impact

Lower (forks for snapshots)

Higher (especially with always mode)

Compact (optimized binary format)

Larger (command log, may contain redundancy)

Recovery Speed

Fast (load optimized snapshot)

Slower (replay all commands)

Backup Friendly

Yes (single compact file)

Less ideal (larger file, may change during backup)

Corruption Risk

Higher (binary format, entire file)

Lower (append-only, partial recovery possible)

Combined Persistence

: For many production environments, using both RDB and AOF provides the best

balance of durability, performance, and recovery options.

Ephemeral Caching

: If Redis is used only as a cache with no data persistence requirements, you can disable

both mechanisms to maximize performance.

Memory-constrained Environments

: In environments with limited RAM, frequent RDB snapshots might cause

performance issues; consider using AOF with less frequent rewrites.

Data Integrity

: For scenarios where data integrity is critical, use AOF with

appendfsync always

,

but be prepared for performance impacts.

When using both RDB and AOF, Redis prioritizes AOF during recovery since it typically contains more recent data.

However, if the AOF file is corrupted, Redis will fall back to the RDB file.

Redis allows you to use both RDB and AOF persistence simultaneously, combining their advantages:

With this hybrid approach:

AOF provides better data safety with minimal data loss window.

RDB provides compact backups and faster recovery if AOF becomes corrupted.

Redis 4.0+ can automatically use RDB format as the base of AOF files, improving efficiency.

Since Redis 7.0, a new "Multi-part AOF" persistence mechanism has been introduced, which stores both RDB snapshots and

incremental AOF files, providing better performance and recoverability.

1
2# Enable both RDB snapshots and AOF
3save 900 1
4save 300 10
5save 60 10000
6appendonly yes
7appendfsync everysec
8        

Redis replication allows you to create copies of a Redis instance (replicas) that maintain the same dataset as the

master instance. This provides several benefits, including read scaling, data redundancy, and geographical distribution.

Redis uses a master-replica replication model:

: The primary Redis instance that accepts write operations and propagates changes to replicas.

: Secondary Redis instances that maintain a copy of the master's dataset.

Replicas can accept read queries but not writes (by default).

Replication Architecture Diagram

Redis replication is simple to configure. On the replica server:

When a replica connects to its master, it performs a full synchronization:

Master creates a point-in-time snapshot (RDB file) of its dataset.

Master transfers the RDB file to the replica.

Replica loads the RDB file into memory.

Master sends the commands that were processed during the RDB transfer.

Replica applies these commands to reach the same state as the master.

Master continues to stream commands to the replica in real-time.

To disconnect a replica from its master and promote it to a master:

1
2# In redis.conf of the replica:
3replicaof 192.168.1.100 6379  # Master IP and port
4
5# Or using the REPLICAOF command at runtime:
6REPLICAOF 192.168.1.100 6379
7        

1
2# Convert replica to master
3REPLICAOF NO ONE
4        

Redis offers several configuration options for replication:

Read-only replica

: By default, replicas are read-only; this can be changed but is not recommended:

Authentication

: If the master requires authentication, replicas need to authenticate:

Disk-less replication

: For faster replication, master can stream RDB directly to replicas without writing to disk:

Partial resynchronization

: If connection is lost temporarily, Redis tries to resume replication without full resync:

1
2replica-read-only yes  # Default setting
3            

1
2# On the replica:
3masterauth "your-master-password"
4            

1
2repl-diskless-sync yes
3repl-diskless-sync-delay 5  # Wait for multiple replicas to connect
4            

1
2# Size of replication backlog on master (to support partial resync)
3repl-backlog-size 1mb
4            

Redis supports several replication topologies:

Single Master with Multiple Replicas

(Star topology):

This is the simplest and most common topology, with each replica connecting directly to the master.

Chained Replication

(Cascade topology):

Replicas can have their own replicas, reducing load on the master but increasing replication delay down the chain.

Hierarchical Replication

(Tree topology):

A mix of star and chained topologies, useful for geographical distribution and load balancing.

While Redis allows complex replication topologies, it's important to consider the increased complexity in monitoring

and failure recovery. Each level in the chain adds replication lag and makes failover more challenging.

Learn about Replication Benefits and Use Cases.

Replication allows you to scale read operations by distributing read queries across multiple replicas:

Route write operations to the master.

Distribute read operations across replicas.

Scale horizontally by adding more replicas as read volume increases.

1
2// Example client configuration (pseudo-code)
3const masterClient = createRedisClient({ host: 'master.redis', port: 6379 });
4const replicaPool = [
5  createRedisClient({ host: 'replica1.redis', port: 6379 }),
6  createRedisClient({ host: 'replica2.redis', port: 6379 }),
7  createRedisClient({ host: 'replica3.redis', port: 6379 }),
8];
9
10// Write operations go to master
11function writeData(key, value) {
12  return masterClient.set(key, value);
13}
14
15// Read operations go to replicas (round-robin)
16function readData(key) {
17  const replicaIndex = Math.floor(Math.random() * replicaPool.length);
18  return replicaPool[replicaIndex].get(key);
19}
20        

Replication provides data redundancy, which is a prerequisite for high availability:

If the master fails, a replica can be promoted to become the new master.

This promotion can be manual or automated through Redis Sentinel or Redis Cluster.

Applications can continue to function with minimal downtime during failover.

Replicas can be used for backup operations without affecting the master:

Configure a dedicated backup replica.

on the replica to create an RDB file.

Copy the RDB file to backup storage.

This approach avoids additional load on the master during backup operations.

Replicas can be deployed across different geographical locations for better performance:

Place replicas closer to users to reduce read latency.

Deploy in multiple regions for disaster recovery.

Consider the impact of network latency on replication lag.

Learn about Replication Challenges.

Replication lag is the delay between an operation being executed on the master and being applied on the replica.

It can be caused by network latency, high write load, or resource constraints on the replica.

You can monitor replication lag using the

INFO REPLICATION

To mitigate replication lag:

Ensure replicas have sufficient resources (CPU, memory, network bandwidth).

Consider using diskless replication for faster initial synchronization.

Monitor and alert on excessive lag.

For read-after-write consistency, consider routing related reads to the master.

1
2# On the master
3INFO REPLICATION
4
5# Sample output:
6# role:master
7# connected_slaves:2
8# slave0:ip=192.168.1.101,port=6379,state=online,offset=1234,lag=0
9# slave1:ip=192.168.1.102,port=6379,state=online,offset=1230,lag=1
10        

Split brain is a condition where multiple nodes believe they are the master due to network partition or communication failure.

This can lead to data inconsistency if clients write to different masters.

Redis replication itself doesn't prevent split brain; this is typically addressed using:

Redis Sentinel

: Provides automated failover with quorum-based decision making.

Redis Cluster

: Implements automatic failover with majority agreement.

Manual failover procedures

: Carefully coordinated by operations teams.

Redis replication provides eventual consistency, not strong consistency. This means:

Writes to the master are eventually reflected on replicas, but not immediately.

Reading from replicas might return stale data during replication lag.

Applications must be designed with these consistency limitations in mind.

For applications requiring read-after-write consistency, consider these patterns:

Write to Master, Read from Master

: For operations requiring strong consistency.

Sticky Master

: After writing, direct subsequent reads from the same client to the master for a period.

Client-Side Timestamps

: Track operation time and retry reads if data is too stale.

Redis Sentinel provides monitoring, notification, and automatic failover for Redis deployments. It addresses the high

availability challenges not solved by replication alone.

Sentinel is a distributed system where multiple Sentinel instances work together:

Monitoring

: Sentinels continuously check if master and replica instances are working as expected.

Notification

: Sentinels can notify system administrators about events via API.

Automatic failover

: When a master is not working as expected, Sentinels can start a failover process.

Configuration provider

: Clients connect to Sentinels to find the current master address.

Sentinel Architecture Diagram

Sentinel is configured in a separate configuration file (

sentinel.conf

This configuration:

Monitors a master named "mymaster" at 192.168.1.100:6379

Requires 2 Sentinels to agree that the master is down (quorum)

Considers the master down after 5 seconds of non-responsiveness

Sets the failover process timeout to 60 seconds

Allows only 1 replica to sync with the new master at a time during failover

Start Sentinel processes on multiple servers:

1
2# Basic Sentinel configuration
3sentinel monitor mymaster 192.168.1.100 6379 2
4sentinel down-after-milliseconds mymaster 5000
5sentinel failover-timeout mymaster 60000
6sentinel parallel-syncs mymaster 1
7        

1
2redis-sentinel /path/to/sentinel.conf
3        

When a master is detected as failing, Sentinel initiates the failover process:

: Multiple Sentinels agree the master is down (based on quorum).

Leader election

: Sentinels select a leader among themselves (using Raft algorithm).

Replica selection

: The leader selects the best replica to promote.

: The selected replica is promoted to master.

Reconfiguration

: Other replicas are reconfigured to replicate from the new master.

Client notification

: Sentinels notify clients about the new master.

Applications need to be Sentinel-aware to benefit from automatic failover:

1
2// Example of Sentinel-aware client (Node.js with ioredis)
3const Redis = require('ioredis');
4
5// Connect using Sentinel
6const redis = new Redis({
7  sentinels: [
8    { host: 'sentinel1', port: 26379 },
9    { host: 'sentinel2', port: 26379 },
10    { host: 'sentinel3', port: 26379 }
11  ],
12  name: 'mymaster',  // The name defined in sentinel.conf
13  role: 'master'     // Connect to the master
14});
15
16// Read operations
17redis.get('key').then(result => console.log(result));
18
19// For read scaling, connect to replicas
20const redisReplica = new Redis({
21  sentinels: [
22    { host: 'sentinel1', port: 26379 },
23    { host: 'sentinel2', port: 26379 },
24    { host: 'sentinel3', port: 26379 }
25  ],
26  name: 'mymaster',
27  role: 'slave'      // Connect to replicas
28});
29
30// Read from replica
31redisReplica.get('key').then(result => console.log(result));
32        

Learn about Best Practices for Persistence and Replication.

Use both RDB and AOF

for the best durability and recovery options.

Configure save directives

based on write frequency and durability needs.

Use appendfsync everysec

for a good balance between performance and durability.

Monitor disk space

to avoid running out of space during persistence operations.

Consider performance impacts

of persistence, especially on large datasets.

Implement regular backups

of RDB files for disaster recovery.

Deploy at least one replica

for data redundancy.

Use Sentinel or Cluster

for automated failover.

Monitor replication lag

and set alerts for excessive lag.

Ensure sufficient bandwidth

between master and replicas.

Consider diskless replication

for faster replica synchronization.

Design for eventual consistency

or implement appropriate consistency patterns.

Test failover procedures

regularly to ensure they work as expected.

Regular monitoring

command to check persistence and replication status:

Log monitoring

: Track Redis logs for replication and persistence events.

Performance monitoring

: Watch for latency spikes during RDB saves or AOF rewrites.

Regular testing

: Periodically test recovery from RDB and AOF files.

Failover drills

: Practice master failure and recovery procedures.

1
2# Monitor persistence status
3INFO PERSISTENCE
4
5# Monitor replication status
6INFO REPLICATION
7            

Redis persistence and replication are essential for building reliable, scalable, and high-performance applications.

By understanding these mechanisms and their trade-offs, you can choose the right configuration for your specific

requirements.

Key takeaways from this lesson:

Redis offers two persistence options: RDB (snapshots) and AOF (command log)

RDB provides faster recovery but with a potentially larger data loss window

AOF provides better durability but with higher disk I/O and larger files

Using both mechanisms provides the best combination of durability and recovery speed

Replication allows for read scaling, high availability, and data redundancy

Redis Sentinel addresses the high availability challenges through automated failover

Understanding consistency models is crucial for designing applications with Redis replication

In the next lesson, we'll explore Redis Clustering, which provides both horizontal scaling and high availability

through automatic sharding and replica management.