Terraform State & Backends

INTERMEDIATE

Master Terraform state management, remote backends, and state locking for team collaboration and production environments.

60 min

Tutorial 4 of 7

Learning Objectives

Understand how Terraform state works and why it matters
Configure remote backends for team collaboration
Implement state locking to prevent conflicts
Use state commands for inspection and manipulation
Manage sensitive data in state files
Implement state backup and recovery strategies

Introduction

Terraform state is the source of truth about your infrastructure. Understanding state management is crucial for reliable infrastructure operations, especially in team environments. In this lesson, we'll explore how Terraform tracks resources, configure remote backends, implement locking mechanisms, and follow best practices for production environments.

1. Understanding Terraform State

Terraform state is a JSON file that maps your configuration to real-world resources. When you run terraform apply, Terraform:

State File Contents:

Resource mappings (config → real infrastructure)
Resource metadata and attributes
Dependencies between resources
Provider configurations
Outputs and their values

Why State Matters:

Performance: Caching reduces API calls to cloud providers
Metadata: Stores information not available from APIs
Dependencies: Tracks relationships between resources
Concurrency: Enables locking to prevent conflicts

State File Location:

By default, Terraform stores state in terraform.tfstate in your working directory. This works for solo projects but has serious limitations for teams.

State Inspection:

Bash

1# List all resources in state
2terraform state list
3
4# Show details of specific resource
5terraform state show aws_instance.web
6
7# Pull current state
8terraform state pull > state.json

Key Principle: Never edit state files manually - always use Terraform commands to modify state.

2. Local vs Remote State

Local State (Default):

Stored in local terraform.tfstate file
Fast and simple for single-user projects
Problems:
No collaboration (can't share state)
No locking (concurrent applies cause corruption)
No security (sensitive data in plain text)
Manual backup required

Remote State (Recommended):

Remote backends store state in a shared location accessible to team members.

Popular Remote Backends:

1. Amazon S3 + DynamoDB (Most Common)

State stored in S3 bucket
Locking via DynamoDB table
Encryption at rest
Versioning for rollback

2. Terraform Cloud

Managed by HashiCorp
Built-in locking and versioning
Web UI for state inspection
Free tier available

3. Azure Blob Storage

State stored in Azure Storage
Built-in locking
Azure AD authentication

4. Google Cloud Storage

State in GCS bucket
Built-in locking
IAM-based access control

5. HashiCorp Consul

Distributed state storage
Built-in locking
High availability

3. Configuring S3 Backend

Amazon S3 with DynamoDB is the most popular remote backend setup.

Step 1: Create S3 Bucket (One-Time Setup)

Bash

1# Create S3 bucket for state
2aws s3api create-bucket \
3  --bucket my-terraform-state-bucket \
4  --region us-east-1
5
6# Enable versioning (for rollback)
7aws s3api put-bucket-versioning \
8  --bucket my-terraform-state-bucket \
9  --versioning-configuration Status=Enabled
10
11# Enable encryption
12aws s3api put-bucket-encryption \
13  --bucket my-terraform-state-bucket \
14  --server-side-encryption-configuration '{
15    "Rules": [{
16      "ApplyServerSideEncryptionByDefault": {
17        "SSEAlgorithm": "AES256"
18      }
19    }]
20  }'

Step 2: Create DynamoDB Table for Locking

Bash

1aws dynamodb create-table \
2  --table-name terraform-state-lock \
3  --attribute-definitions AttributeName=LockID,AttributeType=S \
4  --key-schema AttributeName=LockID,KeyType=HASH \
5  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Step 3: Configure Backend in Terraform

HCL

1terraform {
2  backend "s3" {
3    bucket         = "my-terraform-state-bucket"
4    key            = "prod/terraform.tfstate"
5    region         = "us-east-1"
6    encrypt        = true
7    dynamodb_table = "terraform-state-lock"
8  }
9}

Step 4: Initialize Backend

Bash

1terraform init
2# Terraform will prompt to copy local state to S3

4. State Locking

State locking prevents multiple users from modifying infrastructure simultaneously, which could corrupt the state.

How Locking Works:

User A runs terraform apply
Terraform acquires lock (writes to DynamoDB)
User B tries terraform apply
Terraform detects lock, waits or fails
User A completes, releases lock
User B can now proceed

Lock Behavior by Backend:

S3 + DynamoDB:

Automatic locking on apply, destroy, import
Lock stored in DynamoDB with metadata
Configurable timeout

Terraform Cloud:

Automatic locking on all state-modifying operations
Web UI shows who has the lock
Force-unlock capability for admins

Manual Lock Override (Emergency Only):

Bash

1# View lock info
2terraform force-unlock <lock-id>
3
4# Force unlock (use with extreme caution)
5terraform force-unlock -force <lock-id>

Best Practices:

Never force-unlock unless you're absolutely certain
Investigate why lock wasn't released automatically
Ensure no other operations are running
Consider using Terraform Cloud for better lock visibility

5. State Commands

Terraform provides commands to inspect and manipulate state safely.

Inspection Commands:

Bash

1# List all resources
2terraform state list
3
4# Show resource details
5terraform state show aws_instance.web
6
7# Show specific attribute
8terraform state show -json aws_instance.web | jq '.values.public_ip'
9
10# Pull entire state
11terraform state pull > backup.tfstate

Modification Commands:

Bash

1# Remove resource from state (doesn't delete real resource)
2terraform state rm aws_instance.old_server
3
4# Move/rename resource in state
5terraform state mv aws_instance.web aws_instance.web_server
6
7# Replace provider address
8terraform state replace-provider registry.terraform.io/hashicorp/aws \
9  registry.terraform.io/hashicorp/aws

Import Existing Resources:

Bash

1# Import existing AWS instance
2terraform import aws_instance.web i-1234567890abcdef0
3
4# Import with module
5terraform import module.vpc.aws_vpc.main vpc-1234567890abcdef0

Common Use Cases:

Renaming Resources:

Bash

1# 1. Rename in configuration files
2# 2. Move in state
3terraform state mv aws_instance.old_name aws_instance.new_name
4# 3. Apply (should show no changes)
5terraform apply

Removing Abandoned Resources:

Bash

1# Remove from state without destroying
2terraform state rm aws_instance.manual_creation

6. Sensitive Data in State

State files often contain sensitive data like passwords, API keys, and private IP addresses.

What's in State:

Database passwords (if configured)
API keys and tokens
Private keys
Connection strings
Internal IP addresses

Security Measures:

1. Encrypt State at Rest:

HCL

1terraform {
2  backend "s3" {
3    bucket  = "my-state-bucket"
4    key     = "terraform.tfstate"
5    encrypt = true  # Enable S3 encryption
6    kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/..."
7  }
8}

2. Restrict Access:

JSON

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Principal": {
7        "AWS": "arn:aws:iam::123456789012:role/TerraformRole"
8      },
9      "Action": ["s3:GetObject", "s3:PutObject"],
10      "Resource": "arn:aws:s3:::my-state-bucket/*"
11    }
12  ]
13}

3. Use Secrets Management:

Never hardcode secrets in Terraform
Use AWS Secrets Manager, Vault, or similar
Reference secrets dynamically
Mark outputs as sensitive

4. Never Commit State to Git:

.GITIGNORE

1# Terraform
2*.tfstate
3*.tfstate.*
4*.tfstate.backup
5.terraform/

Sensitive Outputs:

HCL

1output "db_password" {
2  value     = aws_db_instance.main.password
3  sensitive = true  # Hide from console output
4}

7. State Backup & Recovery

Proper backup strategies prevent data loss and enable recovery from mistakes.

Automatic Backups:

Local State:

Bash

1# Terraform creates automatic backups
2# terraform.tfstate.backup

S3 Backend:

HCL

1# Enable versioning on S3 bucket
2resource "aws_s3_bucket_versioning" "state" {
3  bucket = aws_s3_bucket.terraform_state.id
4
5  versioning_configuration {
6    status = "Enabled"
7  }
8}
9
10# Lifecycle rule for old versions
11resource "aws_s3_bucket_lifecycle_configuration" "state" {
12  bucket = aws_s3_bucket.terraform_state.id
13
14  rule {
15    id     = "keep-30-versions"
16    status = "Enabled"
17
18    noncurrent_version_expiration {
19      noncurrent_days = 90
20    }
21  }
22}

Manual Backup:

Bash

1# Pull and save state
2terraform state pull > "backup-$(date +%Y%m%d-%H%M%S).tfstate"

Recovery Procedures:

1. Restore from S3 Version:

Bash

1# List versions
2aws s3api list-object-versions \
3  --bucket my-terraform-state-bucket \
4  --prefix prod/terraform.tfstate
5
6# Download specific version
7aws s3api get-object \
8  --bucket my-terraform-state-bucket \
9  --key prod/terraform.tfstate \
10  --version-id <version-id> \
11  restored.tfstate
12
13# Push back to S3
14terraform state push restored.tfstate

2. Disaster Recovery:

Bash

1# If state is lost, rebuild from real infrastructure
2terraform import aws_instance.web i-1234567890abcdef0
3terraform import aws_vpc.main vpc-1234567890abcdef0
4# ... import all resources

Backup Best Practices:

Enable S3 versioning (keeps last 90 versions)
Automate periodic state exports
Test recovery procedures regularly
Document import commands for all resources
Keep backup in separate AWS account

8. Workspace-Based State Management

Workspaces allow multiple state files for the same configuration (e.g., dev, staging, prod).

Workspace Commands:

Bash

1# List workspaces
2terraform workspace list
3
4# Create new workspace
5terraform workspace new staging
6
7# Switch workspace
8terraform workspace select prod
9
10# Current workspace
11terraform workspace show
12
13# Delete workspace
14terraform workspace delete staging

How Workspaces Work:

Each workspace has its own state file
Same configuration, different state
Useful for environment separation

State File Locations:

Local Backend:

Text

1terraform.tfstate.d/
2├── dev/
3│   └── terraform.tfstate
4├── staging/
5│   └── terraform.tfstate
6└── prod/
7    └── terraform.tfstate

S3 Backend:

Text

1my-bucket/
2├── env:/dev/terraform.tfstate
3├── env:/staging/terraform.tfstate
4└── env:/prod/terraform.tfstate

Using Workspace Name in Config:

HCL

1resource "aws_instance" "web" {
2  ami           = "ami-123"
3  instance_type = terraform.workspace == "prod" ? "t3.large" : "t3.micro"
4
5  tags = {
6    Name        = "web-${terraform.workspace}"
7    Environment = terraform.workspace
8  }
9}

Pros:

Single configuration for all environments
Easy to switch between environments
Automatic state isolation

Cons:

Easy to accidentally destroy wrong environment
Limited blast radius protection
Configuration drift between environments

Recommendation: Use workspaces for lower environments (dev/staging), but use separate configurations for production.

Complete Remote Backend Setup

Learn how to configure a production-ready S3 backend with encryption, versioning, and state locking.

Example CodeHTML

1<!DOCTYPE html>
2<html lang="en">
3<head>
4    <meta charset="UTF-8">
5    <meta name="viewport" content="width=device-width, initial-scale=1.0">
6    <title>Terraform Remote Backend Setup</title>
7    <style>
8        body {
9            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
10            line-height: 1.6;
11            max-width: 900px;
12            margin: 0 auto;
13            padding: 20px;
14            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
15        }
16        .container {
17            background: white;
18            padding: 30px;
19            border-radius: 10px;
20            box-shadow: 0 10px 30px rgba(0,0,0,0.3);
21        }
22        h1 { color: #5e35b1; margin-top: 0; }
23        h2 { color: #7e57c2; border-bottom: 2px solid #e0e0e0; padding-bottom: 10px; }
24        .step {
25            background: #f5f5f5;
26            padding: 15px;
27            margin: 15px 0;
28            border-left: 4px solid #5e35b1;
29            border-radius: 4px;
30        }
31        .step h3 { margin-top: 0; color: #5e35b1; }
32        code {
33            background: #263238;
34            color: #aed581;
35            padding: 2px 6px;
36            border-radius: 3px;
37            font-family: 'Courier New', monospace;
38        }
39        .command {
40            background: #263238;
41            color: #aed581;
42            padding: 15px;
43            border-radius: 5px;
44            overflow-x: auto;
45            margin: 10px 0;
46        }
47        .success { color: #4caf50; font-weight: bold; }
48        .warning { color: #ff9800; font-weight: bold; }
49        .checklist {
50            background: #e8f5e9;
51            padding: 15px;
52            border-radius: 5px;
53            margin: 15px 0;
54        }
55        .checklist ul { margin: 10px 0; }
56        .checklist li { margin: 5px 0; }
57    </style>
58</head>
59<body>
60    <div class="container">
61        <h1>🔐 Terraform Remote Backend Setup</h1>
62        <p>Complete guide to configuring a production-ready remote backend with S3, DynamoDB, encryption, and state locking.</p>
63
64        <h2>Prerequisites</h2>
65        <div class="checklist">
66            <strong>Before you begin:</strong>
67            <ul>
68                <li>✓ AWS CLI installed and configured</li>
69                <li>✓ Terraform installed (v1.0+)</li>
70                <li>✓ IAM permissions for S3 and DynamoDB</li>
71                <li>✓ Understanding of Terraform basics</li>
72            </ul>
73        </div>
74
75        <div class="step">
76            <h3>Step 1: Create S3 Bucket for State</h3>
77            <div class="command">
78# Create bucket<br>
79aws s3api create-bucket \<br>
80&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
81&nbsp;&nbsp;--region us-east-1<br><br>
82# Enable versioning (critical for rollback)<br>
83aws s3api put-bucket-versioning \<br>
84&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
85&nbsp;&nbsp;--versioning-configuration Status=Enabled<br><br>
86# Enable encryption<br>
87aws s3api put-bucket-encryption \<br>
88&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
89&nbsp;&nbsp;--server-side-encryption-configuration '{<br>
90&nbsp;&nbsp;&nbsp;&nbsp;"Rules": [{<br>
91&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"ApplyServerSideEncryptionByDefault": {<br>
92&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"SSEAlgorithm": "AES256"<br>
93&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<br>
94&nbsp;&nbsp;&nbsp;&nbsp;}]<br>
95&nbsp;&nbsp;}'<br><br>
96# Block public access<br>
97aws s3api put-public-access-block \<br>
98&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
99&nbsp;&nbsp;--public-access-block-configuration \<br>
100&nbsp;&nbsp;"BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
101            </div>
102            <p class="success">✓ S3 bucket configured with versioning, encryption, and no public access</p>
103        </div>
104
105        <div class="step">
106            <h3>Step 2: Create DynamoDB Table for Locking</h3>
107            <div class="command">
108aws dynamodb create-table \<br>
109&nbsp;&nbsp;--table-name terraform-state-lock \<br>
110&nbsp;&nbsp;--attribute-definitions AttributeName=LockID,AttributeType=S \<br>
111&nbsp;&nbsp;--key-schema AttributeName=LockID,KeyType=HASH \<br>
112&nbsp;&nbsp;--billing-mode PAY_PER_REQUEST
113            </div>
114            <p class="success">✓ DynamoDB table created for state locking</p>
115        </div>
116
117        <div class="step">
118            <h3>Step 3: Configure Backend in Terraform</h3>
119            <p>Create <code>backend.tf</code>:</p>
120            <div class="command">
121terraform {<br>
122&nbsp;&nbsp;backend "s3" {<br>
123&nbsp;&nbsp;&nbsp;&nbsp;bucket         = "my-company-terraform-state"<br>
124&nbsp;&nbsp;&nbsp;&nbsp;key            = "prod/infrastructure.tfstate"<br>
125&nbsp;&nbsp;&nbsp;&nbsp;region         = "us-east-1"<br>
126&nbsp;&nbsp;&nbsp;&nbsp;encrypt        = true<br>
127&nbsp;&nbsp;&nbsp;&nbsp;dynamodb_table = "terraform-state-lock"<br>
128&nbsp;&nbsp;}<br>
129}
130            </div>
131        </div>
132
133        <div class="step">
134            <h3>Step 4: Initialize and Migrate State</h3>
135            <div class="command">
136# Initialize backend<br>
137terraform init<br><br>
138# Terraform will prompt:<br>
139# "Do you want to copy existing state to the new backend?"<br>
140# Answer: yes
141            </div>
142            <p class="success">✓ State migrated to remote backend</p>
143        </div>
144
145        <div class="step">
146            <h3>Step 5: Verify Remote State</h3>
147            <div class="command">
148# Pull state from S3<br>
149terraform state pull<br><br>
150# List resources<br>
151terraform state list<br><br>
152# Check S3<br>
153aws s3 ls s3://my-company-terraform-state/prod/
154            </div>
155        </div>
156
157        <h2>Testing State Locking</h2>
158        <div class="step">
159            <h3>Simulate Concurrent Access</h3>
160            <p><strong>Terminal 1:</strong></p>
161            <div class="command">
162terraform apply -auto-approve<br>
163# Holds lock during apply
164            </div>
165            <p><strong>Terminal 2 (while Terminal 1 is running):</strong></p>
166            <div class="command">
167terraform plan<br>
168# Will wait for lock or timeout
169            </div>
170            <p class="success">✓ Locking prevents concurrent modifications</p>
171        </div>
172
173        <h2>Best Practices Checklist</h2>
174        <div class="checklist">
175            <strong>Production Readiness:</strong>
176            <ul>
177                <li>✓ S3 versioning enabled</li>
178                <li>✓ Encryption at rest enabled</li>
179                <li>✓ DynamoDB locking configured</li>
180                <li>✓ Public access blocked</li>
181                <li>✓ IAM permissions restricted</li>
182                <li>✓ Lifecycle rules for old versions</li>
183                <li>✓ State backup procedures documented</li>
184                <li>✓ .gitignore includes *.tfstate*</li>
185            </ul>
186        </div>
187
188        <h2>Common Issues & Solutions</h2>
189        <div class="step">
190            <p><span class="warning">⚠ Error: "Error acquiring state lock"</span></p>
191            <p><strong>Cause:</strong> Another user or process has the lock, or previous operation crashed.</p>
192            <p><strong>Solution:</strong></p>
193            <div class="command">
194# Check DynamoDB for lock<br>
195aws dynamodb scan --table-name terraform-state-lock<br><br>
196# If stale, force unlock (use cautiously)<br>
197terraform force-unlock &lt;lock-id&gt;
198            </div>
199        </div>
200
201        <div class="step">
202            <p><span class="warning">⚠ Error: "Failed to load state"</span></p>
203            <p><strong>Cause:</strong> State file corrupted or inaccessible.</p>
204            <p><strong>Solution:</strong></p>
205            <div class="command">
206# Restore from S3 version<br>
207aws s3api list-object-versions \<br>
208&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
209&nbsp;&nbsp;--prefix prod/infrastructure.tfstate<br><br>
210# Download specific version<br>
211aws s3api get-object \<br>
212&nbsp;&nbsp;--bucket my-company-terraform-state \<br>
213&nbsp;&nbsp;--key prod/infrastructure.tfstate \<br>
214&nbsp;&nbsp;--version-id &lt;version-id&gt; \<br>
215&nbsp;&nbsp;restored.tfstate
216            </div>
217        </div>
218
219        <h2>Next Steps</h2>
220        <ul>
221            <li>Configure IAM policies for team access</li>
222            <li>Set up workspace-based state management</li>
223            <li>Implement automated state backups</li>
224            <li>Consider Terraform Cloud for enhanced features</li>
225            <li>Document state recovery procedures for your team</li>
226        </ul>
227    </div>
228</body>
229</html>

How It Works

This example provides a complete, production-ready remote backend setup with:

S3 Configuration: Versioning, encryption, and public access blocking
DynamoDB Locking: Prevents concurrent state modifications
Backend Configuration: Proper Terraform backend setup
Migration Process: Step-by-step state migration from local to remote
Testing: How to verify locking works correctly
Troubleshooting: Common issues and their solutions

The setup ensures team collaboration, prevents state corruption, and maintains security best practices for production infrastructure.

Key Takeaways

Terraform state is the source of truth mapping configuration to real infrastructure
Remote backends enable team collaboration with shared state storage
State locking (via DynamoDB) prevents concurrent modifications and corruption
Always enable S3 versioning for state rollback capability
Encrypt state at rest and restrict access - state contains sensitive data
Use terraform state commands for safe state inspection and manipulation
Never edit state files manually or commit them to version control
Implement automated backup procedures and test recovery regularly

What's Next?

Your progress is saved in this browser, on this device only.

Previous: Terraform Providers & Resources

Next: Terraform Modules

DevOps

Terraform State & Backends

Learning Objectives

Introduction

1. Understanding Terraform State

2. Local vs Remote State

3. Configuring S3 Backend

4. State Locking

5. State Commands

6. Sensitive Data in State

7. State Backup & Recovery

8. Workspace-Based State Management

Complete Remote Backend Setup

How It Works

Key Takeaways

What's Next?

Table of Contents