Master continuous delivery and deployment practices including release strategies, deployment patterns, rollback procedures, and production monitoring for reliable software delivery
Continuous Delivery (CD) extends CI by ensuring code is always in a deployable state, while Continuous Deployment automatically releases every change that passes all stages of the production pipeline. Together, these practices enable teams to deliver value to users faster with higher quality and lower risk.
Understanding CD is crucial because it:
The key difference between Continuous Delivery and Continuous Deployment is the final step:
Modern CD practices involve sophisticated deployment strategies, infrastructure automation, comprehensive monitoring, and automated rollback capabilities. This lesson covers the essential concepts and hands-on techniques for implementing robust CD pipelines.
Continuous Delivery (CD)
Continuous Delivery means your code is always in a deployable state and can be released to production at any time with a manual decision. This is ideal for:
Key characteristics:
Continuous Deployment
Continuous Deployment takes automation one step further—every change that passes all tests is automatically deployed to production. This is ideal for:
Key characteristics:
The Deployment Pipeline Stages:
1Source → Build → Test → Stage → Approve → Deploy → Monitor
2 ↑
3 Manual (CD)
4 or
5 Automated (Continuous Deployment)Example: Continuous Delivery Pipeline
1# GitHub Actions workflow with manual approval
2name: Continuous Delivery Pipeline
3
4on:
5 push:
6 branches: [main]
7
8jobs:
9 # Automated stages
10 build-test-stage:
11 runs-on: ubuntu-latest
12 steps:
13 - uses: actions/checkout@v4
14 - run: npm ci
15 - run: npm test
16 - run: npm run build
17 - run: deploy-to-staging.sh
18
19 # Manual approval required
20 deploy-production:
21 needs: build-test-stage
22 environment:
23 name: production
24 url: https://example.com
25 runs-on: ubuntu-latest
26 steps:
27 - run: deploy-to-production.shIn this example, staging deployment is automatic, but production requires manual approval through GitHub's environment protection rules.
Decision Framework:
Choose Continuous Delivery when:
✓ You need human oversight for deployments
✓ Compliance requires approval trails
✓ You have scheduled release windows
✓ You're building automation maturity
Choose Continuous Deployment when:
✓ You have comprehensive test coverage (>80%)
✓ You have robust monitoring and alerting
✓ You can quickly rollback or fix forward
✓ Your team embraces high deployment frequency
Many organizations start with Continuous Delivery and evolve to Continuous Deployment as their testing, monitoring, and team maturity increases.
Modern deployment strategies minimize risk and downtime by carefully controlling how new versions are introduced to production.
1. Blue-Green Deployment
Maintain two identical production environments (Blue and Green). Deploy to the inactive environment, test it, then switch traffic.
Advantages:
Disadvantages:
When to use:
1# Blue-Green deployment process
2# 1. Deploy to green environment
3kubectl apply -f deployment-green.yaml
4
5# 2. Wait for green to be healthy
6kubectl rollout status deployment/app-green
7
8# 3. Run tests against green
9npm run test:smoke -- --url=https://green.example.com
10
11# 4. Switch traffic from blue to green
12kubectl patch service app -p '{"spec":{"selector":{"version":"green"}}}'
13
14# 5. Monitor for issues
15# If problems occur, switch back to blue immediately
16kubectl patch service app -p '{"spec":{"selector":{"version":"blue"}}}'2. Canary Deployment
Gradually roll out changes to a small subset of users before full deployment. Named after coal miners using canaries to detect toxic gas.
Advantages:
Disadvantages:
When to use:
1# Canary deployment process
2# 1. Deploy canary version to 5% of traffic
3kubectl apply -f deployment-canary.yaml
4kubectl patch service app -p '{"spec":{"traffic":{"canary":5,"stable":95}}}'
5
6# 2. Monitor canary metrics for 15 minutes
7# - Error rate
8# - Response time
9# - Business metrics
10
11# 3. If metrics look good, increase to 25%
12kubectl patch service app -p '{"spec":{"traffic":{"canary":25,"stable":75}}}'
13
14# 4. Continue increasing: 50%, 75%, 100%
15# OR rollback if issues detected
16kubectl delete deployment app-canary3. Rolling Update
Gradually replace instances of the old version with the new version, one at a time or in batches.
Advantages:
Disadvantages:
When to use:
1# Kubernetes rolling update
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: app
6spec:
7 replicas: 10
8 strategy:
9 type: RollingUpdate
10 rollingUpdate:
11 maxSurge: 2 # Allow 2 extra pods during update
12 maxUnavailable: 1 # Keep most pods available
13 template:
14 spec:
15 containers:
16 - name: app
17 image: app:v2.0.0
18 readinessProbe: # Only route traffic when ready
19 httpGet:
20 path: /health
21 port: 8080
22 initialDelaySeconds: 5
23 periodSeconds: 54. Recreate Deployment
Stop all old versions, then start new versions. Simplest strategy with downtime.
When to use:
5. A/B Testing Deployment
Similar to canary but routes traffic based on user attributes, not random percentage.
When to use:
Strategy Comparison:
| Strategy | Downtime | Complexity | Rollback | Infrastructure |
|----------|----------|------------|----------|---------------|
| Blue-Green | None | Low | Instant | 2x |
| Canary | None | High | Fast | 1.1-1.5x |
| Rolling | None | Medium | Slow | 1x |
| Recreate | Yes | Very Low | Slow | 1x |
| A/B Testing | None | Very High | Fast | 1.1-2x |
Release automation streamlines the deployment process while maintaining necessary controls through approval workflows, release gates, and automated validation.
Release Pipeline Components
Automated Release Process:
1# Complete automated release workflow
2name: Automated Release
3
4on:
5 push:
6 tags:
7 - 'v*' # Trigger on version tags
8
9jobs:
10 # Create release artifacts
11 build-artifacts:
12 runs-on: ubuntu-latest
13 steps:
14 - uses: actions/checkout@v4
15
16 - name: Extract version from tag
17 id: version
18 run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
19
20 - name: Build application
21 run: npm run build
22
23 - name: Create distribution archive
24 run: |
25 tar -czf app-${{ steps.version.outputs.VERSION }}.tar.gz dist/
26
27 - name: Generate checksums
28 run: sha256sum app-${{ steps.version.outputs.VERSION }}.tar.gz > checksums.txt
29
30 - name: Sign artifacts
31 run: |
32 gpg --armor --detach-sign app-${{ steps.version.outputs.VERSION }}.tar.gz
33
34 - name: Upload artifacts
35 uses: actions/upload-artifact@v4
36 with:
37 name: release-artifacts
38 path: |
39 app-*.tar.gz
40 app-*.tar.gz.asc
41 checksums.txt
42
43 # Generate changelog
44 generate-changelog:
45 runs-on: ubuntu-latest
46 steps:
47 - uses: actions/checkout@v4
48 with:
49 fetch-depth: 0 # Need full history for changelog
50
51 - name: Generate changelog
52 run: |
53 npx conventional-changelog-cli -p angular -i CHANGELOG.md -s
54
55 - name: Upload changelog
56 uses: actions/upload-artifact@v4
57 with:
58 name: changelog
59 path: CHANGELOG.md
60
61 # Automated quality gates
62 quality-gates:
63 runs-on: ubuntu-latest
64 steps:
65 - uses: actions/checkout@v4
66
67 # Quality gate 1: Test coverage
68 - name: Check test coverage
69 run: |
70 npm test -- --coverage
71 COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
72 if (( $(echo "$COVERAGE < 80" | bc -l) )); then
73 echo "Coverage $COVERAGE% is below 80% threshold"
74 exit 1
75 fi
76
77 # Quality gate 2: Code quality
78 - name: Run code quality checks
79 run: |
80 npm run lint
81 npm run type-check
82
83 # Quality gate 3: Security scan
84 - name: Security vulnerability scan
85 run: |
86 npm audit --audit-level=high
87 npx snyk test --severity-threshold=high
88
89 # Quality gate 4: Performance benchmarks
90 - name: Run performance tests
91 run: |
92 npm run test:performance
93 # Check that performance meets baselines
94
95 # Create GitHub release (requires passing gates)
96 create-release:
97 needs: [build-artifacts, generate-changelog, quality-gates]
98 runs-on: ubuntu-latest
99 permissions:
100 contents: write
101 steps:
102 - uses: actions/download-artifact@v4
103
104 - name: Create GitHub Release
105 uses: softprops/action-gh-release@v1
106 with:
107 name: Release ${{ github.ref_name }}
108 body_path: changelog/CHANGELOG.md
109 files: |
110 release-artifacts/app-*.tar.gz
111 release-artifacts/app-*.tar.gz.asc
112 release-artifacts/checksums.txt
113 draft: false
114 prerelease: false
115
116 # Deploy to staging (automatic)
117 deploy-staging:
118 needs: create-release
119 uses: ./.github/workflows/deploy.yml
120 with:
121 environment: staging
122 version: ${{ github.ref_name }}
123 secrets: inherit
124
125 # Deploy to production (manual approval)
126 deploy-production:
127 needs: deploy-staging
128 uses: ./.github/workflows/deploy.yml
129 with:
130 environment: production
131 version: ${{ github.ref_name }}
132 secrets: inheritApproval Workflow Patterns:
Change Advisory Board (CAB) Automation:
1// Automated CAB approval system
2interface ChangeRequest {
3 id: string
4 title: string
5 risk: 'low' | 'medium' | 'high'
6 type: 'feature' | 'bugfix' | 'hotfix'
7 tests: TestResults
8 security: SecurityScanResults
9}
10
11interface ApprovalPolicy {
12 evaluate(change: ChangeRequest): Promise<ApprovalDecision>
13}
14
15class RiskBasedApprovalPolicy implements ApprovalPolicy {
16 async evaluate(change: ChangeRequest): Promise<ApprovalDecision> {
17 // Low risk with passing tests: Auto-approve
18 if (
19 change.risk === 'low' &&
20 change.tests.coverage > 80 &&
21 change.tests.passed &&
22 change.security.vulnerabilities === 0
23 ) {
24 return {
25 approved: true,
26 automatic: true,
27 reason: 'Low risk change with full test coverage'
28 }
29 }
30
31 // Hotfixes: Expedited approval
32 if (change.type === 'hotfix') {
33 return {
34 approved: false,
35 requiresApprovers: ['oncall-engineer', 'team-lead'],
36 reason: 'Hotfix requires expedited approval'
37 }
38 }
39
40 // High risk: Multiple approvals
41 if (change.risk === 'high') {
42 return {
43 approved: false,
44 requiresApprovers: ['engineering-manager', 'product-owner', 'sre-lead'],
45 reason: 'High risk change requires multiple approvals'
46 }
47 }
48
49 // Default: Standard approval
50 return {
51 approved: false,
52 requiresApprovers: ['team-lead'],
53 reason: 'Standard approval process'
54 }
55 }
56}Best Practices:
✅ Automate the boring stuff: Version bumping, changelog generation, artifact creation
✅ Make approvals meaningful: Require approvals for production, but not for staging
✅ Use quality gates: Enforce standards through automation, not just process
✅ Audit everything: Log all approvals, deployments, and rollbacks
✅ Enable self-service: Let developers deploy to dev/staging without approvals
✅ Time-box approvals: Auto-reject or escalate stale approvals
✅ Separate duties: Different people/teams for different approval stages
Infrastructure as Code (IaC) treats infrastructure configuration as software code, enabling version control, code review, automated testing, and reproducible deployments.
Why Infrastructure as Code?
Traditional manual infrastructure management leads to:
IaC solves these problems by:
IaC Tools:
Terraform Deployment Example:
1# main.tf - Complete deployment infrastructure
2
3terraform {
4 required_version = ">= 1.0"
5
6 backend "s3" {
7 bucket = "terraform-state-bucket"
8 key = "production/terraform.tfstate"
9 region = "us-east-1"
10
11 dynamodb_table = "terraform-locks"
12 encrypt = true
13 }
14
15 required_providers {
16 aws = {
17 source = "hashicorp/aws"
18 version = "~> 5.0"
19 }
20 }
21}
22
23provider "aws" {
24 region = var.aws_region
25
26 default_tags {
27 tags = {
28 Environment = var.environment
29 Project = "my-app"
30 ManagedBy = "terraform"
31 }
32 }
33}
34
35# Variables
36variable "environment" {
37 description = "Environment name"
38 type = string
39}
40
41variable "app_version" {
42 description = "Application version to deploy"
43 type = string
44}
45
46variable "instance_count" {
47 description = "Number of application instances"
48 type = number
49 default = 3
50}
51
52# VPC and networking
53module "vpc" {
54 source = "terraform-aws-modules/vpc/aws"
55
56 name = "${var.environment}-vpc"
57 cidr = "10.0.0.0/16"
58
59 azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
60 private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
61 public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
62
63 enable_nat_gateway = true
64 enable_vpn_gateway = false
65
66 tags = {
67 Name = "${var.environment}-vpc"
68 }
69}
70
71# Application Load Balancer
72resource "aws_lb" "app" {
73 name = "${var.environment}-app-lb"
74 internal = false
75 load_balancer_type = "application"
76 security_groups = [aws_security_group.alb.id]
77 subnets = module.vpc.public_subnets
78
79 enable_deletion_protection = var.environment == "production"
80
81 tags = {
82 Name = "${var.environment}-app-lb"
83 }
84}
85
86# Target group for blue deployment
87resource "aws_lb_target_group" "blue" {
88 name = "${var.environment}-app-blue"
89 port = 8080
90 protocol = "HTTP"
91 vpc_id = module.vpc.vpc_id
92
93 health_check {
94 enabled = true
95 healthy_threshold = 2
96 unhealthy_threshold = 3
97 timeout = 5
98 interval = 30
99 path = "/health"
100 matcher = "200"
101 }
102
103 deregistration_delay = 30
104}
105
106# Target group for green deployment
107resource "aws_lb_target_group" "green" {
108 name = "${var.environment}-app-green"
109 port = 8080
110 protocol = "HTTP"
111 vpc_id = module.vpc.vpc_id
112
113 health_check {
114 enabled = true
115 healthy_threshold = 2
116 unhealthy_threshold = 3
117 timeout = 5
118 interval = 30
119 path = "/health"
120 matcher = "200"
121 }
122
123 deregistration_delay = 30
124}
125
126# ALB listener with blue-green traffic splitting
127resource "aws_lb_listener" "app" {
128 load_balancer_arn = aws_lb.app.arn
129 port = 443
130 protocol = "HTTPS"
131 ssl_policy = "ELBSecurityPolicy-TLS-1-2-2017-01"
132 certificate_arn = aws_acm_certificate.app.arn
133
134 default_action {
135 type = "forward"
136
137 forward {
138 target_group {
139 arn = aws_lb_target_group.blue.arn
140 weight = var.traffic_distribution.blue
141 }
142
143 target_group {
144 arn = aws_lb_target_group.green.arn
145 weight = var.traffic_distribution.green
146 }
147
148 stickiness {
149 enabled = true
150 duration = 3600
151 }
152 }
153 }
154}
155
156# ECS Cluster
157resource "aws_ecs_cluster" "app" {
158 name = "${var.environment}-app-cluster"
159
160 setting {
161 name = "containerInsights"
162 value = "enabled"
163 }
164}
165
166# ECS Task Definition
167resource "aws_ecs_task_definition" "app" {
168 family = "${var.environment}-app"
169 network_mode = "awsvpc"
170 requires_compatibilities = ["FARGATE"]
171 cpu = "1024"
172 memory = "2048"
173 execution_role_arn = aws_iam_role.ecs_execution.arn
174 task_role_arn = aws_iam_role.ecs_task.arn
175
176 container_definitions = jsonencode([
177 {
178 name = "app"
179 image = "myregistry.io/app:${var.app_version}"
180 essential = true
181
182 portMappings = [
183 {
184 containerPort = 8080
185 protocol = "tcp"
186 }
187 ]
188
189 environment = [
190 {
191 name = "ENVIRONMENT"
192 value = var.environment
193 },
194 {
195 name = "VERSION"
196 value = var.app_version
197 }
198 ]
199
200 secrets = [
201 {
202 name = "DATABASE_URL"
203 valueFrom = aws_secretsmanager_secret.database_url.arn
204 }
205 ]
206
207 logConfiguration = {
208 logDriver = "awslogs"
209 options = {
210 "awslogs-group" = aws_cloudwatch_log_group.app.name
211 "awslogs-region" = var.aws_region
212 "awslogs-stream-prefix" = "app"
213 }
214 }
215
216 healthCheck = {
217 command = ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
218 interval = 30
219 timeout = 5
220 retries = 3
221 startPeriod = 60
222 }
223 }
224 ])
225}
226
227# ECS Service (Blue)
228resource "aws_ecs_service" "blue" {
229 name = "${var.environment}-app-blue"
230 cluster = aws_ecs_cluster.app.id
231 task_definition = aws_ecs_task_definition.app.arn
232 desired_count = var.traffic_distribution.blue > 0 ? var.instance_count : 0
233 launch_type = "FARGATE"
234
235 network_configuration {
236 subnets = module.vpc.private_subnets
237 security_groups = [aws_security_group.app.id]
238 assign_public_ip = false
239 }
240
241 load_balancer {
242 target_group_arn = aws_lb_target_group.blue.arn
243 container_name = "app"
244 container_port = 8080
245 }
246
247 deployment_configuration {
248 maximum_percent = 200
249 minimum_healthy_percent = 100
250 }
251
252 enable_execute_command = true
253}
254
255# ECS Service (Green)
256resource "aws_ecs_service" "green" {
257 name = "${var.environment}-app-green"
258 cluster = aws_ecs_cluster.app.id
259 task_definition = aws_ecs_task_definition.app.arn
260 desired_count = var.traffic_distribution.green > 0 ? var.instance_count : 0
261 launch_type = "FARGATE"
262
263 network_configuration {
264 subnets = module.vpc.private_subnets
265 security_groups = [aws_security_group.app.id]
266 assign_public_ip = false
267 }
268
269 load_balancer {
270 target_group_arn = aws_lb_target_group.green.arn
271 container_name = "app"
272 container_port = 8080
273 }
274
275 deployment_configuration {
276 maximum_percent = 200
277 minimum_healthy_percent = 100
278 }
279
280 enable_execute_command = true
281}
282
283# Outputs
284output "load_balancer_dns" {
285 value = aws_lb.app.dns_name
286}
287
288output "ecs_cluster_name" {
289 value = aws_ecs_cluster.app.name
290}GitOps Workflow:
GitOps uses Git as the single source of truth for infrastructure and application deployments:
Best Practices:
✅ Use modules: Create reusable infrastructure components
✅ Separate environments: Different state files for dev/staging/production
✅ Lock state: Use remote state locking to prevent concurrent modifications
✅ Plan before apply: Always review changes before applying
✅ Automate in CI/CD: Integrate IaC into deployment pipelines
✅ Tag resources: Use consistent tagging for cost tracking and management
✅ Secrets management: Never commit secrets; use secret managers
Effective deployment monitoring and rollback procedures are essential for maintaining system reliability and quickly recovering from issues.
Monitoring Pyramid:
1 /\
2 / \ Business Metrics
3 /----\ (conversion rate, revenue)
4 / \
5 /--------\ Application Metrics
6 / \ (error rate, latency, throughput)
7 /------------\
8 / \ Infrastructure Metrics
9 /----------------\ (CPU, memory, disk, network)
10/------------------\1. Infrastructure Monitoring
Monitor the underlying infrastructure health:
2. Application Monitoring
Track application-level metrics:
These are the "Golden Signals" from Google's SRE book.
3. Business Monitoring
Monitor business-critical metrics:
Comprehensive Monitoring Stack:
1# Prometheus + Grafana monitoring setup
2# docker-compose.yml
3
4version: '3.8'
5
6services:
7 # Prometheus for metrics collection
8 prometheus:
9 image: prom/prometheus:latest
10 volumes:
11 - ./prometheus.yml:/etc/prometheus/prometheus.yml
12 - prometheus-data:/prometheus
13 command:
14 - '--config.file=/etc/prometheus/prometheus.yml'
15 - '--storage.tsdb.retention.time=30d'
16 ports:
17 - "9090:9090"
18
19 # Grafana for visualization
20 grafana:
21 image: grafana/grafana:latest
22 volumes:
23 - grafana-data:/var/lib/grafana
24 - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
25 - ./grafana/datasources:/etc/grafana/provisioning/datasources
26 environment:
27 - GF_SECURITY_ADMIN_PASSWORD=secure_password
28 - GF_USERS_ALLOW_SIGN_UP=false
29 ports:
30 - "3000:3000"
31
32 # Alertmanager for alert routing
33 alertmanager:
34 image: prom/alertmanager:latest
35 volumes:
36 - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
37 ports:
38 - "9093:9093"
39
40 # Node exporter for infrastructure metrics
41 node-exporter:
42 image: prom/node-exporter:latest
43 ports:
44 - "9100:9100"
45
46 # Loki for log aggregation
47 loki:
48 image: grafana/loki:latest
49 volumes:
50 - ./loki.yml:/etc/loki/loki.yml
51 - loki-data:/loki
52 command: -config.file=/etc/loki/loki.yml
53 ports:
54 - "3100:3100"
55
56volumes:
57 prometheus-data:
58 grafana-data:
59 loki-data:Alerting Rules:
1# prometheus-alerts.yml
2groups:
3 - name: deployment_alerts
4 interval: 30s
5 rules:
6 # High error rate alert
7 - alert: HighErrorRate
8 expr: |
9 (
10 sum(rate(http_requests_total{status=~"5.."}[5m]))
11 /
12 sum(rate(http_requests_total[5m]))
13 ) > 0.05
14 for: 2m
15 labels:
16 severity: critical
17 component: application
18 annotations:
19 summary: "High error rate detected"
20 description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
21
22 # High latency alert
23 - alert: HighLatency
24 expr: |
25 histogram_quantile(
26 0.95,
27 rate(http_request_duration_seconds_bucket[5m])
28 ) > 1
29 for: 5m
30 labels:
31 severity: warning
32 component: application
33 annotations:
34 summary: "High request latency"
35 description: "95th percentile latency is {{ $value }}s (threshold: 1s)"
36
37 # Low availability alert
38 - alert: ServiceDown
39 expr: up{job="app"} == 0
40 for: 1m
41 labels:
42 severity: critical
43 component: infrastructure
44 annotations:
45 summary: "Service is down"
46 description: "{{ $labels.instance }} is down"
47
48 # High CPU usage
49 - alert: HighCPUUsage
50 expr: |
51 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
52 for: 10m
53 labels:
54 severity: warning
55 component: infrastructure
56 annotations:
57 summary: "High CPU usage"
58 description: "CPU usage is {{ $value }}% on {{ $labels.instance }}"
59
60 # Deployment anomaly detection
61 - alert: DeploymentAnomalyDetected
62 expr: |
63 (
64 rate(http_requests_total{status=~"5.."}[5m])
65 /
66 rate(http_requests_total{status=~"5.."}[5m] offset 1h)
67 ) > 2
68 for: 3m
69 labels:
70 severity: critical
71 component: deployment
72 annotations:
73 summary: "Deployment anomaly detected"
74 description: "Error rate increased by {{ $value }}x after deployment"Automated Rollback:
1#!/bin/bash
2# automated-rollback.sh
3# Monitors deployment health and automatically rolls back on issues
4
5set -euo pipefail
6
7DEPLOYMENT_NAME="app"
8NAMESPACE="production"
9MONITORING_DURATION=300 # 5 minutes
10CHECK_INTERVAL=30 # 30 seconds
11
12# Prometheus query endpoint
13PROMETHEUS_URL="http://prometheus:9090"
14
15echo "🔍 Monitoring deployment health for ${MONITORING_DURATION}s..."
16
17# Get current revision before deployment
18CURRENT_REVISION=$(kubectl rollout history deployment/$DEPLOYMENT_NAME -n $NAMESPACE | tail -1 | awk '{print $1}')
19
20# Function to check error rate
21check_error_rate() {
22 ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
23 --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[2m])) / sum(rate(http_requests_total[2m]))' \
24 | jq -r '.data.result[0].value[1]')
25
26 echo "$ERROR_RATE"
27}
28
29# Function to check latency
30check_latency() {
31 P95_LATENCY=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
32 --data-urlencode 'query=histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[2m]))' \
33 | jq -r '.data.result[0].value[1]')
34
35 echo "$P95_LATENCY"
36}
37
38# Function to perform rollback
39perform_rollback() {
40 echo "❌ Health check failed! Initiating rollback..."
41
42 # Rollback to previous revision
43 kubectl rollout undo deployment/$DEPLOYMENT_NAME -n $NAMESPACE
44
45 # Wait for rollback to complete
46 kubectl rollout status deployment/$DEPLOYMENT_NAME -n $NAMESPACE
47
48 # Send alert
49 curl -X POST "$SLACK_WEBHOOK_URL" \
50 -H 'Content-Type: application/json' \
51 -d '{
52 "text": "🚨 ALERT: Automatic rollback performed for deployment '"$DEPLOYMENT_NAME"'",
53 "attachments": [{
54 "color": "danger",
55 "text": "Deployment failed health checks and was automatically rolled back to previous version"
56 }]
57 }'
58
59 echo "✅ Rollback completed"
60 exit 1
61}
62
63# Monitor deployment for specified duration
64START_TIME=$(date +%s)
65CHECKS_FAILED=0
66
67while true; do
68 CURRENT_TIME=$(date +%s)
69 ELAPSED=$((CURRENT_TIME - START_TIME))
70
71 if [ $ELAPSED -ge $MONITORING_DURATION ]; then
72 echo "✅ Monitoring completed. Deployment is healthy!"
73 break
74 fi
75
76 # Check error rate
77 ERROR_RATE=$(check_error_rate)
78 if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
79 echo "⚠️ High error rate detected: $ERROR_RATE"
80 ((CHECKS_FAILED++))
81 else
82 echo "✓ Error rate OK: $ERROR_RATE"
83 fi
84
85 # Check latency
86 P95_LATENCY=$(check_latency)
87 if (( $(echo "$P95_LATENCY > 1.0" | bc -l) )); then
88 echo "⚠️ High latency detected: ${P95_LATENCY}s"
89 ((CHECKS_FAILED++))
90 else
91 echo "✓ Latency OK: ${P95_LATENCY}s"
92 fi
93
94 # Rollback if 3 consecutive checks fail
95 if [ $CHECKS_FAILED -ge 3 ]; then
96 perform_rollback
97 fi
98
99 # Reset counter if checks pass
100 if [ $CHECKS_FAILED -gt 0 ] && (( $(echo "$ERROR_RATE <= 0.05" | bc -l) )) && (( $(echo "$P95_LATENCY <= 1.0" | bc -l) )); then
101 CHECKS_FAILED=0
102 fi
103
104 # Wait before next check
105 sleep $CHECK_INTERVAL
106done
107
108# Send success notification
109curl -X POST "$SLACK_WEBHOOK_URL" \
110 -H 'Content-Type: application/json' \
111 -d '{
112 "text": "✅ Deployment successful: '"$DEPLOYMENT_NAME"'",
113 "attachments": [{
114 "color": "good",
115 "text": "All health checks passed during monitoring period"
116 }]
117 }'Rollback Decision Framework:
1Decision Tree for Rollback:
2
3High Error Rate (>5%) ──────────> ROLLBACK
4 │
5 ↓ No
6High Latency (p95 >1s) ────────> ROLLBACK
7 │
8 ↓ No
9Service Unavailable ───────────> ROLLBACK
10 │
11 ↓ No
12Memory/CPU Critical ───────────> INVESTIGATE
13 │ │
14 ↓ No ↓
15Continue Monitoring Manual DecisionBest Practices:
✅ Monitor continuously: Set up dashboards that are always visible
✅ Alert on symptoms, not causes: Alert on user-facing issues first
✅ Reduce alert fatigue: Only alert on actionable items
✅ Use SLOs: Define Service Level Objectives and alert when at risk
✅ Automate rollbacks: For critical issues, automate the rollback process
✅ Practice rollbacks: Regularly test rollback procedures in staging
✅ Keep audit logs: Log all deployments, rollbacks, and manual interventions
✅ Gradual rollouts: Use canary or blue-green to limit blast radius
End-to-end continuous delivery pipeline with automated testing, blue-green deployment, health monitoring, and automated rollback
1# .github/workflows/complete-cd-pipeline.yml
2# Production-ready CD pipeline demonstrating all key concepts
3
4name: Complete CD Pipeline
5
6on:
7 push:
8 branches: [main]
9 tags:
10 - 'v*'
11 workflow_dispatch:
12 inputs:
13 environment:
14 description: 'Environment to deploy'
15 required: true
16 type: choice
17 options: [staging, production]
18 skip_tests:
19 description: 'Skip tests (emergency only)'
20 type: boolean
21 default: false
22
23env:
24 NODE_VERSION: '20'
25 DOCKER_REGISTRY: ghcr.io
26 IMAGE_NAME: ${{ github.repository }}
27
28permissions:
29 contents: read
30 packages: write
31 deployments: write
32 id-token: write
33
34jobs:
35 # ==================== STAGE 1: BUILD & TEST ====================
36
37 build-and-test:
38 name: Build and Test
39 runs-on: ubuntu-latest
40 if: ${{ !inputs.skip_tests }}
41
42 steps:
43 - uses: actions/checkout@v4
44
45 - name: Setup Node.js
46 uses: actions/setup-node@v4
47 with:
48 node-version: ${{ env.NODE_VERSION }}
49 cache: 'npm'
50
51 - name: Install dependencies
52 run: npm ci
53
54 - name: Run linter
55 run: npm run lint
56
57 - name: Type check
58 run: npm run type-check
59
60 - name: Run unit tests
61 run: npm run test:unit -- --coverage
62
63 - name: Check coverage threshold
64 run: |
65 COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
66 if (( $(echo "$COVERAGE < 80" | bc -l) )); then
67 echo "❌ Coverage $COVERAGE% is below 80% threshold"
68 exit 1
69 fi
70 echo "✅ Coverage: $COVERAGE%"
71
72 - name: Run integration tests
73 run: npm run test:integration
74
75 - name: Security scan
76 run: |
77 npm audit --audit-level=high
78 npx snyk test --severity-threshold=high
79
80 - name: Build application
81 run: npm run build
82
83 - name: Upload build artifacts
84 uses: actions/upload-artifact@v4
85 with:
86 name: build-artifacts
87 path: |
88 dist/
89 package.json
90 package-lock.json
91 retention-days: 7
92
93 # ==================== STAGE 2: BUILD CONTAINER ====================
94
95 build-container:
96 name: Build Container Image
97 runs-on: ubuntu-latest
98 needs: build-and-test
99 if: ${{ !failure() }}
100 outputs:
101 image-tag: ${{ steps.meta.outputs.tags }}
102 image-digest: ${{ steps.build.outputs.digest }}
103
104 steps:
105 - uses: actions/checkout@v4
106
107 - name: Set up Docker Buildx
108 uses: docker/setup-buildx-action@v3
109
110 - name: Log in to Container Registry
111 uses: docker/login-action@v3
112 with:
113 registry: ${{ env.DOCKER_REGISTRY }}
114 username: ${{ github.actor }}
115 password: ${{ secrets.GITHUB_TOKEN }}
116
117 - name: Extract metadata
118 id: meta
119 uses: docker/metadata-action@v5
120 with:
121 images: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}
122 tags: |
123 type=ref,event=branch
124 type=semver,pattern={{version}}
125 type=semver,pattern={{major}}.{{minor}}
126 type=sha,prefix=${{github.ref_name}}-
127
128 - name: Download build artifacts
129 uses: actions/download-artifact@v4
130 with:
131 name: build-artifacts
132 path: dist/
133
134 - name: Build and push container
135 id: build
136 uses: docker/build-push-action@v5
137 with:
138 context: .
139 push: true
140 tags: ${{ steps.meta.outputs.tags }}
141 labels: ${{ steps.meta.outputs.labels }}
142 cache-from: type=gha
143 cache-to: type=gha,mode=max
144 platforms: linux/amd64,linux/arm64
145
146 - name: Scan container image
147 uses: aquasecurity/trivy-action@master
148 with:
149 image-ref: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
150 format: 'sarif'
151 output: 'trivy-results.sarif'
152 severity: 'CRITICAL,HIGH'
153 exit-code: '1'
154
155 # ==================== STAGE 3: DEPLOY TO STAGING ====================
156
157 deploy-staging:
158 name: Deploy to Staging
159 runs-on: ubuntu-latest
160 needs: build-container
161 environment:
162 name: staging
163 url: https://staging.example.com
164
165 steps:
166 - uses: actions/checkout@v4
167
168 - name: Configure AWS credentials
169 uses: aws-actions/configure-aws-credentials@v4
170 with:
171 role-to-assume: ${{ secrets.AWS_ROLE_STAGING }}
172 aws-region: us-east-1
173
174 - name: Setup kubectl
175 uses: azure/setup-kubectl@v3
176
177 - name: Configure kubectl
178 run: |
179 aws eks update-kubeconfig --name staging-cluster --region us-east-1
180
181 # Blue-Green Deployment
182 - name: Determine active environment
183 id: active
184 run: |
185 ACTIVE=$(kubectl get service app -n staging -o jsonpath='{.spec.selector.version}')
186 if [ "$ACTIVE" == "blue" ]; then
187 echo "active=blue" >> $GITHUB_OUTPUT
188 echo "inactive=green" >> $GITHUB_OUTPUT
189 else
190 echo "active=green" >> $GITHUB_OUTPUT
191 echo "inactive=blue" >> $GITHUB_OUTPUT
192 fi
193 echo "Active environment: $ACTIVE"
194
195 - name: Deploy to inactive environment
196 run: |
197 INACTIVE=${{ steps.active.outputs.inactive }}
198 IMAGE="${{ needs.build-container.outputs.image-tag }}"
199
200 echo "Deploying to $INACTIVE environment..."
201
202 # Update deployment with new image
203 kubectl set image deployment/app-$INACTIVE \
204 app=$IMAGE \
205 -n staging
206
207 # Wait for rollout
208 kubectl rollout status deployment/app-$INACTIVE \
209 -n staging \
210 --timeout=5m
211
212 - name: Run smoke tests
213 run: |
214 INACTIVE=${{ steps.active.outputs.inactive }}
215
216 # Get service URL for inactive environment
217 INACTIVE_URL="https://${INACTIVE}-staging.example.com"
218
219 echo "Running smoke tests against $INACTIVE_URL..."
220
221 # Health check
222 curl -f "$INACTIVE_URL/health" || exit 1
223
224 # E2E smoke tests
225 npm run test:smoke -- --base-url=$INACTIVE_URL || exit 1
226
227 - name: Switch traffic to new environment
228 run: |
229 INACTIVE=${{ steps.active.outputs.inactive }}
230
231 echo "Switching traffic to $INACTIVE environment..."
232
233 # Update service selector
234 kubectl patch service app \
235 -n staging \
236 -p '{"spec":{"selector":{"version":"'$INACTIVE'"}}}'
237
238 echo "Traffic switched successfully!"
239
240 - name: Monitor deployment health
241 run: |
242 echo "Monitoring deployment for 5 minutes..."
243
244 # Run health monitoring script
245 ./scripts/monitor-deployment.sh \
246 --deployment app \
247 --namespace staging \
248 --duration 300 \
249 --prometheus-url http://prometheus:9090
250
251 - name: Scale down old environment
252 if: success()
253 run: |
254 ACTIVE=${{ steps.active.outputs.active }}
255
256 echo "Scaling down old $ACTIVE environment..."
257 kubectl scale deployment/app-$ACTIVE \
258 -n staging \
259 --replicas=0
260
261 # ==================== STAGE 4: DEPLOY TO PRODUCTION ====================
262
263 deploy-production:
264 name: Deploy to Production
265 runs-on: ubuntu-latest
266 needs: [build-container, deploy-staging]
267 if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')
268 environment:
269 name: production
270 url: https://example.com
271
272 steps:
273 - uses: actions/checkout@v4
274
275 - name: Configure AWS credentials
276 uses: aws-actions/configure-aws-credentials@v4
277 with:
278 role-to-assume: ${{ secrets.AWS_ROLE_PRODUCTION }}
279 aws-region: us-east-1
280
281 - name: Setup kubectl
282 uses: azure/setup-kubectl@v3
283
284 - name: Configure kubectl
285 run: |
286 aws eks update-kubeconfig --name production-cluster --region us-east-1
287
288 # Create deployment record
289 - name: Create deployment record
290 id: deployment
291 uses: chrnorm/deployment-action@v2
292 with:
293 token: ${{ secrets.GITHUB_TOKEN }}
294 environment: production
295 description: Deploying ${{ needs.build-container.outputs.image-tag }}
296
297 # Blue-Green Deployment (same as staging)
298 - name: Determine active environment
299 id: active
300 run: |
301 ACTIVE=$(kubectl get service app -n production -o jsonpath='{.spec.selector.version}')
302 if [ "$ACTIVE" == "blue" ]; then
303 echo "active=blue" >> $GITHUB_OUTPUT
304 echo "inactive=green" >> $GITHUB_OUTPUT
305 else
306 echo "active=green" >> $GITHUB_OUTPUT
307 echo "inactive=blue" >> $GITHUB_OUTPUT
308 fi
309
310 - name: Deploy to inactive environment
311 run: |
312 INACTIVE=${{ steps.active.outputs.inactive }}
313 IMAGE="${{ needs.build-container.outputs.image-tag }}"
314
315 kubectl set image deployment/app-$INACTIVE \
316 app=$IMAGE \
317 -n production
318
319 kubectl rollout status deployment/app-$INACTIVE \
320 -n production \
321 --timeout=10m
322
323 - name: Run production smoke tests
324 run: |
325 INACTIVE=${{ steps.active.outputs.inactive }}
326 INACTIVE_URL="https://${INACTIVE}.example.com"
327
328 curl -f "$INACTIVE_URL/health" || exit 1
329 npm run test:smoke -- --base-url=$INACTIVE_URL || exit 1
330
331 # Gradual traffic shift (canary-style)
332 - name: Shift 10% traffic to new environment
333 run: |
334 ACTIVE=${{ steps.active.outputs.active }}
335 INACTIVE=${{ steps.active.outputs.inactive }}
336
337 kubectl patch service app -n production -p '{
338 "spec": {
339 "traffic": [
340 {"revisionName": "app-'$ACTIVE'", "percent": 90},
341 {"revisionName": "app-'$INACTIVE'", "percent": 10}
342 ]
343 }
344 }'
345
346 echo "10% traffic shifted. Monitoring..."
347 sleep 120
348
349 - name: Shift 50% traffic
350 run: |
351 ACTIVE=${{ steps.active.outputs.active }}
352 INACTIVE=${{ steps.active.outputs.inactive }}
353
354 # Check metrics before proceeding
355 ./scripts/check-canary-metrics.sh || exit 1
356
357 kubectl patch service app -n production -p '{
358 "spec": {
359 "traffic": [
360 {"revisionName": "app-'$ACTIVE'", "percent": 50},
361 {"revisionName": "app-'$INACTIVE'", "percent": 50}
362 ]
363 }
364 }'
365
366 echo "50% traffic shifted. Monitoring..."
367 sleep 180
368
369 - name: Complete traffic shift
370 run: |
371 INACTIVE=${{ steps.active.outputs.inactive }}
372
373 # Final metrics check
374 ./scripts/check-canary-metrics.sh || exit 1
375
376 # Switch all traffic
377 kubectl patch service app -n production -p '{
378 "spec": {
379 "selector": {"version": "'$INACTIVE'"}
380 }
381 }'
382
383 echo "100% traffic shifted to $INACTIVE"
384
385 - name: Monitor production deployment
386 run: |
387 ./scripts/monitor-deployment.sh \
388 --deployment app \
389 --namespace production \
390 --duration 600 \
391 --prometheus-url http://prometheus:9090 \
392 --rollback-on-failure
393
394 - name: Update deployment status (success)
395 if: success()
396 uses: chrnorm/deployment-status@v2
397 with:
398 token: ${{ secrets.GITHUB_TOKEN }}
399 deployment-id: ${{ steps.deployment.outputs.deployment_id }}
400 state: success
401 environment-url: https://example.com
402
403 - name: Update deployment status (failure)
404 if: failure()
405 uses: chrnorm/deployment-status@v2
406 with:
407 token: ${{ secrets.GITHUB_TOKEN }}
408 deployment-id: ${{ steps.deployment.outputs.deployment_id }}
409 state: failure
410
411 - name: Scale down old environment
412 if: success()
413 run: |
414 ACTIVE=${{ steps.active.outputs.active }}
415 kubectl scale deployment/app-$ACTIVE -n production --replicas=0
416
417 - name: Notify team
418 if: always()
419 run: |
420 STATUS=${{ job.status }}
421 COLOR=$([ "$STATUS" == "success" ] && echo "good" || echo "danger")
422 EMOJI=$([ "$STATUS" == "success" ] && echo "✅" || echo "❌")
423
424 curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
425 -H 'Content-Type: application/json' \
426 -d '{
427 "text": "'"$EMOJI"' Production deployment '"$STATUS"'",
428 "attachments": [{
429 "color": "'"$COLOR"'",
430 "fields": [
431 {"title": "Environment", "value": "production", "short": true},
432 {"title": "Version", "value": "'${{ needs.build-container.outputs.image-tag }}'", "short": true},
433 {"title": "Deployed by", "value": "'${{ github.actor }}'", "short": true},
434 {"title": "Status", "value": "'"$STATUS"'", "short": true}
435 ]
436 }]
437 }'Your progress is saved in this browser, on this device only.