Master Docker volumes and data persistence strategies to manage stateful applications, share data between containers, and implement reliable backup solutions.
By default, all data inside a container is lost when the container is removed. Docker volumes provide a mechanism to persist data beyond the container lifecycle, share data between containers, and maintain stateful applications. Understanding Docker storage options is essential for running databases, file storage, and any application that needs to retain data. This lesson covers volumes, bind mounts, tmpfs mounts, and best practices for managing persistent data in containerized environments.
Docker provides three main options for storing data:
Storage Types Comparison:
1┌─────────────────────────────────────────────────────────────────┐
2│ Docker Storage Options │
3├─────────────────────────────────────────────────────────────────┤
4│ │
5│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
6│ │ Volumes │ │ Bind Mounts │ │ tmpfs │ │
7│ │ │ │ │ │ │ │
8│ │ Managed by │ │ Host path │ │ Memory only │ │
9│ │ Docker │ │ mounted │ │ (Linux) │ │
10│ │ │ │ │ │ │ │
11│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
12│ │ │ │ │
13│ ▼ ▼ ▼ │
14│ ┌─────────────────────────────────────────────────────────┐ │
15│ │ Container │ │
16│ │ ┌─────────────────────────────────────────────────────┐│ │
17│ │ │ Container Filesystem ││ │
18│ │ │ /var/lib/data /app/src /tmp/cache ││ │
19│ │ └─────────────────────────────────────────────────────┘│ │
20│ └─────────────────────────────────────────────────────────┘ │
21│ │
22└─────────────────────────────────────────────────────────────────┘1. Volumes (Recommended for Persistence):
1# Created and managed by Docker
2docker volume create mydata
3
4# Stored in Docker's storage area
5# Linux: /var/lib/docker/volumes/
6# Works on all platforms
7
8# Benefits:
9# - Easy backup and migration
10# - Can be shared between containers
11# - Volume drivers enable remote storage
12# - Content can be pre-populated2. Bind Mounts (Host Directory Mapping):
1# Maps host directory to container path
2docker run -v /host/path:/container/path myapp
3
4# Benefits:
5# - Direct access to host files
6# - Good for development (hot reloading)
7# - Uses existing host directories
8
9# Drawbacks:
10# - Depends on host directory structure
11# - Permission issues possible
12# - Less portable3. tmpfs Mounts (Memory-Based):
1# Stored in host memory only (Linux)
2docker run --tmpfs /tmp myapp
3
4# Benefits:
5# - Fast access (RAM speed)
6# - Secure (data never written to disk)
7# - Good for sensitive temporary data
8
9# Drawbacks:
10# - Data lost on container stop
11# - Limited by available RAM
12# - Linux onlyChoosing the Right Option:
| Use Case | Recommended |
|----------|-------------|
| Database storage | Volumes |
| Development hot reload | Bind mounts |
| Sensitive temp data | tmpfs |
| Sharing between containers | Volumes |
| Backup/restore workflows | Volumes |
| CI/CD artifacts | Bind mounts |
Volumes are the preferred mechanism for persisting data in Docker:
Creating and Managing Volumes:
1# Create a named volume
2docker volume create mydata
3
4# Create with options
5docker volume create --driver local \
6 --opt type=none \
7 --opt device=/path/to/data \
8 --opt o=bind \
9 mydata
10
11# List all volumes
12docker volume ls
13
14# Inspect volume details
15docker volume inspect mydata
16# Output:
17# [
18# {
19# "CreatedAt": "2024-01-15T10:30:00Z",
20# "Driver": "local",
21# "Labels": {},
22# "Mountpoint": "/var/lib/docker/volumes/mydata/_data",
23# "Name": "mydata",
24# "Options": {},
25# "Scope": "local"
26# }
27# ]
28
29# Remove a volume
30docker volume rm mydata
31
32# Remove all unused volumes
33docker volume prune
34
35# Remove all unused volumes (including anonymous)
36docker volume prune -aUsing Volumes with Containers:
1# Mount volume using -v (short syntax)
2docker run -d -v mydata:/var/lib/mysql mysql:8
3
4# Mount volume using --mount (long syntax, recommended)
5docker run -d \
6 --mount source=mydata,target=/var/lib/mysql \
7 mysql:8
8
9# Read-only volume
10docker run -d \
11 --mount source=mydata,target=/data,readonly \
12 myapp
13
14# Anonymous volume (Docker assigns name)
15docker run -d -v /var/lib/mysql mysql:8
16
17# Multiple volumes
18docker run -d \
19 -v dbdata:/var/lib/postgresql/data \
20 -v dblogs:/var/log/postgresql \
21 postgres:15Volume Mount Options:
1# Full mount syntax
2--mount type=volume,source=mydata,target=/data,readonly,volume-opt=key=value
3
4# Options:
5# type: volume, bind, or tmpfs
6# source (src): Volume name or host path
7# target (dst, destination): Container path
8# readonly (ro): Mount as read-only
9# volume-opt: Driver-specific optionsPre-populating Volumes:
1# In Dockerfile - VOLUME instruction
2FROM nginx:alpine
3COPY html/ /usr/share/nginx/html/
4VOLUME /usr/share/nginx/html
5
6# When volume is mounted to empty container path,
7# existing data is copied to volumeBind mounts map host directories directly into containers, perfect for development workflows:
Basic Bind Mount Usage:
1# Mount current directory
2docker run -v $(pwd):/app myapp
3
4# Windows PowerShell
5docker run -v ${PWD}:/app myapp
6
7# Using --mount syntax (more explicit)
8docker run --mount type=bind,source=$(pwd),target=/app myapp
9
10# Read-only bind mount
11docker run -v $(pwd)/config:/app/config:ro myappDevelopment Workflow Example:
1# Node.js development with hot reload
2docker run -d \
3 --name dev-app \
4 -v $(pwd)/src:/app/src \
5 -v $(pwd)/package.json:/app/package.json \
6 -v /app/node_modules \
7 -p 3000:3000 \
8 node:18-alpine npm run dev
9
10# Explanation:
11# -v $(pwd)/src:/app/src # Source code (hot reload)
12# -v $(pwd)/package.json:... # Package file
13# -v /app/node_modules # Anonymous volume for node_modules
14# # Prevents overwriting container's node_modulesHandling Permissions:
1# Run container as current user
2docker run -u $(id -u):$(id -g) -v $(pwd):/app myapp
3
4# Fix permission issues on Linux
5docker run \
6 --user $(id -u):$(id -g) \
7 -v $(pwd):/app \
8 myapp
9
10# In Dockerfile, match host user
11ARG USER_ID=1000
12ARG GROUP_ID=1000
13RUN groupadd -g $GROUP_ID appgroup && \
14 useradd -u $USER_ID -g appgroup appuser
15USER appuserBind Mount Propagation:
1# Propagation modes control how mounts are shared
2# between container and host
3
4# rprivate (default): No propagation
5docker run -v /host/path:/container/path:rprivate myapp
6
7# rshared: Mount changes visible both ways
8docker run -v /host/path:/container/path:rshared myapp
9
10# rslave: Host changes visible in container
11docker run -v /host/path:/container/path:rslave myappSELinux Considerations (RHEL/CentOS):
1# Add :z for shared content
2docker run -v /host/path:/container/path:z myapp
3
4# Add :Z for private unshared content
5docker run -v /host/path:/container/path:Z myappVolumes enable data sharing between multiple containers:
Shared Volume Pattern:
1# Create shared volume
2docker volume create shared-data
3
4# Container 1: Write data
5docker run -d \
6 --name writer \
7 -v shared-data:/data \
8 alpine sh -c 'while true; do date >> /data/log.txt; sleep 5; done'
9
10# Container 2: Read data
11docker run -d \
12 --name reader \
13 -v shared-data:/data:ro \
14 alpine sh -c 'tail -f /data/log.txt'
15
16# Verify sharing
17docker logs readerMulti-Container Application:
1# Web server + Log processor sharing logs
2docker volume create app-logs
3
4# Web server writes logs
5docker run -d \
6 --name webserver \
7 -v app-logs:/var/log/nginx \
8 -p 80:80 \
9 nginx:alpine
10
11# Log processor reads logs
12docker run -d \
13 --name log-processor \
14 -v app-logs:/logs:ro \
15 alpine sh -c 'tail -f /logs/access.log | grep -v healthcheck'Volumes-From Pattern:
1# Create data container
2docker create --name data-container \
3 -v /data \
4 alpine /bin/true
5
6# Use volumes from data container
7docker run -d --volumes-from data-container myapp
8
9# Multiple containers share the same volumes
10docker run -d --volumes-from data-container another-appSidecar Container Pattern:
1# Main application container
2docker run -d \
3 --name app \
4 -v app-data:/data \
5 myapp
6
7# Sidecar for backup
8docker run -d \
9 --name backup-sidecar \
10 -v app-data:/source:ro \
11 -v backup-dest:/backup \
12 alpine sh -c '
13 while true; do
14 cp -r /source/* /backup/
15 echo "Backup completed at $(date)"
16 sleep 3600
17 done
18 'Database Cluster Sharing:
1# Shared volume for database cluster
2docker volume create pg-data
3
4# Primary database
5docker run -d \
6 --name pg-primary \
7 -v pg-data:/var/lib/postgresql/data \
8 -e POSTGRES_PASSWORD=secret \
9 postgres:15
10
11# Note: For actual replication, use proper
12# PostgreSQL streaming replication, not shared volumesImplementing reliable backup and restore procedures for Docker volumes:
Backup Using Temporary Container:
1# Backup volume to tar archive
2docker run --rm \
3 -v mydata:/source:ro \
4 -v $(pwd):/backup \
5 alpine tar czf /backup/mydata-backup.tar.gz -C /source .
6
7# With timestamp
8docker run --rm \
9 -v mydata:/source:ro \
10 -v $(pwd):/backup \
11 alpine tar czf /backup/mydata-$(date +%Y%m%d-%H%M%S).tar.gz -C /source .Restore from Backup:
1# Create new volume
2docker volume create mydata-restored
3
4# Restore from tar archive
5docker run --rm \
6 -v mydata-restored:/target \
7 -v $(pwd):/backup:ro \
8 alpine tar xzf /backup/mydata-backup.tar.gz -C /targetDatabase-Specific Backup:
1# PostgreSQL backup
2docker exec pg-container pg_dump -U postgres mydb > backup.sql
3
4# PostgreSQL backup with compression
5docker exec pg-container pg_dump -U postgres -Fc mydb > backup.dump
6
7# MySQL backup
8docker exec mysql-container mysqldump -u root -p mydb > backup.sql
9
10# MongoDB backup
11docker exec mongo-container mongodump --out /backup
12docker cp mongo-container:/backup ./mongo-backupAutomated Backup Script:
1#!/bin/bash
2# automated-backup.sh
3
4BACKUP_DIR="/backups"
5RETENTION_DAYS=7
6VOLUME_NAME="mydata"
7DATE=$(date +%Y%m%d-%H%M%S)
8
9# Create backup
10echo "Starting backup of $VOLUME_NAME..."
11docker run --rm \
12 -v $VOLUME_NAME:/source:ro \
13 -v $BACKUP_DIR:/backup \
14 alpine tar czf /backup/$VOLUME_NAME-$DATE.tar.gz -C /source .
15
16# Verify backup
17if [ $? -eq 0 ]; then
18 echo "Backup successful: $VOLUME_NAME-$DATE.tar.gz"
19
20 # Remove old backups
21 find $BACKUP_DIR -name "$VOLUME_NAME-*.tar.gz" -mtime +$RETENTION_DAYS -delete
22 echo "Old backups cleaned up"
23else
24 echo "Backup failed!"
25 exit 1
26fi
27
28# Optional: Upload to S3
29# aws s3 cp $BACKUP_DIR/$VOLUME_NAME-$DATE.tar.gz s3://mybucket/backups/Volume Clone:
1# Clone a volume
2docker volume create mydata-clone
3
4docker run --rm \
5 -v mydata:/source:ro \
6 -v mydata-clone:/dest \
7 alpine sh -c 'cp -av /source/. /dest/'Offsite Backup with Restic:
1# Using restic for encrypted backups
2docker run --rm \
3 -v mydata:/data:ro \
4 -v restic-cache:/root/.cache/restic \
5 -e RESTIC_REPOSITORY=s3:s3.amazonaws.com/mybucket \
6 -e AWS_ACCESS_KEY_ID=xxx \
7 -e AWS_SECRET_ACCESS_KEY=xxx \
8 -e RESTIC_PASSWORD=secret \
9 restic/restic backup /dataDeploy a production-ready PostgreSQL database with proper volume management, backup automation, and monitoring.
1#!/bin/bash
2# Production Database Deployment with Docker Volumes
3# Complete setup with backup, monitoring, and high availability
4
5set -e
6
7echo "=============================================="
8echo "Production Database Deployment"
9echo "=============================================="
10
11# Configuration
12DB_NAME="production_db"
13DB_USER="app_user"
14DB_PASSWORD="secure_password_123"
15BACKUP_RETENTION_DAYS=30
16NETWORK_NAME="db-network"
17
18# ==========================================
19# Step 1: Create Infrastructure
20# ==========================================
21setup_infrastructure() {
22 echo "\n=== Setting Up Infrastructure ==="
23
24 # Create network
25 docker network create $NETWORK_NAME 2>/dev/null || true
26
27 # Create volumes
28 docker volume create pg-data
29 docker volume create pg-backups
30 docker volume create pg-wal-archive
31
32 echo "Volumes created:"
33 docker volume ls --filter "name=pg-"
34}
35
36# ==========================================
37# Step 2: Deploy PostgreSQL
38# ==========================================
39deploy_postgres() {
40 echo "\n=== Deploying PostgreSQL ==="
41
42 # Create custom PostgreSQL configuration
43 mkdir -p /tmp/pg-config
44 cat > /tmp/pg-config/postgresql.conf << 'EOF'
45# Connection Settings
46listen_addresses = '*'
47max_connections = 200
48
49# Memory Settings
50shared_buffers = 256MB
51effective_cache_size = 768MB
52work_mem = 4MB
53maintenance_work_mem = 64MB
54
55# WAL Settings
56wal_level = replica
57archive_mode = on
58archive_command = 'cp %p /var/lib/postgresql/wal-archive/%f'
59max_wal_senders = 3
60wal_keep_size = 1GB
61
62# Logging
63logging_collector = on
64log_directory = 'pg_log'
65log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
66log_statement = 'ddl'
67log_min_duration_statement = 1000
68
69# Performance
70random_page_cost = 1.1
71effective_io_concurrency = 200
72EOF
73
74 cat > /tmp/pg-config/pg_hba.conf << 'EOF'
75# TYPE DATABASE USER ADDRESS METHOD
76local all all trust
77host all all 127.0.0.1/32 scram-sha-256
78host all all ::1/128 scram-sha-256
79host all all 0.0.0.0/0 scram-sha-256
80host replication all 0.0.0.0/0 scram-sha-256
81EOF
82
83 # Start PostgreSQL container
84 docker run -d \
85 --name postgres-primary \
86 --network $NETWORK_NAME \
87 --restart unless-stopped \
88 -e POSTGRES_USER=$DB_USER \
89 -e POSTGRES_PASSWORD=$DB_PASSWORD \
90 -e POSTGRES_DB=$DB_NAME \
91 -e PGDATA=/var/lib/postgresql/data/pgdata \
92 -v pg-data:/var/lib/postgresql/data \
93 -v pg-wal-archive:/var/lib/postgresql/wal-archive \
94 -v /tmp/pg-config/postgresql.conf:/etc/postgresql/postgresql.conf:ro \
95 -v /tmp/pg-config/pg_hba.conf:/etc/postgresql/pg_hba.conf:ro \
96 -p 5432:5432 \
97 --health-cmd="pg_isready -U $DB_USER -d $DB_NAME" \
98 --health-interval=10s \
99 --health-timeout=5s \
100 --health-retries=5 \
101 postgres:15-alpine \
102 postgres -c 'config_file=/etc/postgresql/postgresql.conf' -c 'hba_file=/etc/postgresql/pg_hba.conf'
103
104 echo "Waiting for PostgreSQL to be ready..."
105 until docker exec postgres-primary pg_isready -U $DB_USER; do
106 sleep 2
107 done
108 echo "PostgreSQL is ready!"
109}
110
111# ==========================================
112# Step 3: Setup Backup Container
113# ==========================================
114setup_backup() {
115 echo "\n=== Setting Up Backup System ==="
116
117 # Create backup script
118 cat > /tmp/backup-script.sh << 'BACKUP_SCRIPT'
119#!/bin/sh
120set -e
121
122BACKUP_DIR="/backups"
123DATE=$(date +%Y%m%d-%H%M%S)
124RETENTION_DAYS=${RETENTION_DAYS:-30}
125
126echo "Starting backup at $(date)"
127
128# Full backup using pg_dump
129PGPASSWORD=$DB_PASSWORD pg_dump \
130 -h postgres-primary \
131 -U $DB_USER \
132 -d $DB_NAME \
133 -Fc \
134 -f "$BACKUP_DIR/full-$DATE.dump"
135
136# Verify backup
137pg_restore --list "$BACKUP_DIR/full-$DATE.dump" > /dev/null
138
139echo "Backup completed: full-$DATE.dump"
140
141# Cleanup old backups
142find $BACKUP_DIR -name "full-*.dump" -mtime +$RETENTION_DAYS -delete
143echo "Old backups cleaned up"
144
145# List current backups
146echo "Current backups:"
147ls -lh $BACKUP_DIR/*.dump 2>/dev/null || echo "No backups found"
148BACKUP_SCRIPT
149
150 chmod +x /tmp/backup-script.sh
151
152 # Run backup container
153 docker run -d \
154 --name postgres-backup \
155 --network $NETWORK_NAME \
156 --restart unless-stopped \
157 -e DB_USER=$DB_USER \
158 -e DB_PASSWORD=$DB_PASSWORD \
159 -e DB_NAME=$DB_NAME \
160 -e RETENTION_DAYS=$BACKUP_RETENTION_DAYS \
161 -v pg-backups:/backups \
162 -v /tmp/backup-script.sh:/backup.sh:ro \
163 postgres:15-alpine \
164 sh -c 'while true; do /backup.sh; sleep 86400; done'
165
166 echo "Backup container started (daily backups)"
167}
168
169# ==========================================
170# Step 4: Setup Monitoring
171# ==========================================
172setup_monitoring() {
173 echo "\n=== Setting Up Monitoring ==="
174
175 # PostgreSQL Exporter for Prometheus
176 docker run -d \
177 --name postgres-exporter \
178 --network $NETWORK_NAME \
179 --restart unless-stopped \
180 -e DATA_SOURCE_NAME="postgresql://$DB_USER:$DB_PASSWORD@postgres-primary:5432/$DB_NAME?sslmode=disable" \
181 -p 9187:9187 \
182 quay.io/prometheuscommunity/postgres-exporter
183
184 echo "PostgreSQL Exporter running on port 9187"
185}
186
187# ==========================================
188# Step 5: Verify Deployment
189# ==========================================
190verify_deployment() {
191 echo "\n=== Verifying Deployment ==="
192
193 echo "\nContainers:"
194 docker ps --filter "name=postgres" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
195
196 echo "\nVolumes:"
197 docker volume ls --filter "name=pg-"
198
199 echo "\nVolume sizes:"
200 for vol in pg-data pg-backups pg-wal-archive; do
201 size=$(docker run --rm -v $vol:/data alpine du -sh /data 2>/dev/null | cut -f1)
202 echo " $vol: $size"
203 done
204
205 echo "\nDatabase connection test:"
206 docker exec postgres-primary psql -U $DB_USER -d $DB_NAME -c "SELECT version();"
207
208 echo "\nHealth status:"
209 docker inspect postgres-primary --format '{{.State.Health.Status}}'
210}
211
212# ==========================================
213# Step 6: Utility Functions
214# ==========================================
215manual_backup() {
216 echo "\n=== Running Manual Backup ==="
217 docker exec postgres-backup /backup.sh
218}
219
220restore_backup() {
221 local backup_file=$1
222 if [ -z "$backup_file" ]; then
223 echo "Usage: restore_backup <backup_file>"
224 docker run --rm -v pg-backups:/backups alpine ls /backups/
225 return 1
226 fi
227
228 echo "Restoring from $backup_file..."
229 docker exec postgres-primary pg_restore \
230 -U $DB_USER \
231 -d $DB_NAME \
232 --clean \
233 --if-exists \
234 "/backups/$backup_file"
235
236 echo "Restore completed!"
237}
238
239cleanup() {
240 echo "\n=== Cleanup ==="
241 docker rm -f postgres-primary postgres-backup postgres-exporter 2>/dev/null || true
242 # Uncomment to remove volumes:
243 # docker volume rm pg-data pg-backups pg-wal-archive 2>/dev/null || true
244 docker network rm $NETWORK_NAME 2>/dev/null || true
245 echo "Cleanup complete!"
246}
247
248# ==========================================
249# Main Execution
250# ==========================================
251main() {
252 setup_infrastructure
253 deploy_postgres
254 setup_backup
255 setup_monitoring
256 verify_deployment
257
258 echo "\n=============================================="
259 echo "Deployment Complete!"
260 echo "=============================================="
261 echo "
262PostgreSQL: localhost:5432
263 Database: $DB_NAME
264 User: $DB_USER
265
266Metrics: http://localhost:9187/metrics
267
268Commands:
269 manual_backup # Run manual backup
270 restore_backup <file> # Restore from backup
271 cleanup # Remove all containers
272
273Volumes:
274 pg-data # Database files
275 pg-backups # Backup archives
276 pg-wal-archive # WAL archives
277"
278}
279
280# Run main function
281mainYour progress is saved in this browser, on this device only.