Master Deep RL: Deep Q-Networks (DQN), Policy Gradients, Actor-Critic methods, and Proximal Policy Optimization (PPO). Learn how neural networks enable RL to scale to high-dimensional problems.
Deep Reinforcement Learning combines the decision-making power of RL with the representation learning capabilities of deep neural networks. This breakthrough enables agents to learn directly from high-dimensional sensory inputs like images and handle complex continuous action spaces.
In 2013, DeepMind's DQN learned to play 49 Atari games from raw pixels—matching human performance without game-specific engineering. Since then, deep RL has powered AlphaGo defeating world champions, OpenAI's Dota 2 bots, robotic manipulation, and autonomous vehicles.
In October 2025, deep RL continues to advance: robots learning dexterous manipulation, AI agents mastering real-time strategy games, autonomous systems navigating complex environments, and foundation models fine-tuned with RL from human feedback (RLHF)—the technique behind ChatGPT and Claude.
This lesson covers the essential deep RL algorithms you need to build intelligent agents that learn from pixels, control continuous systems, and master complex tasks.
The Scaling Problem with tabular RL:
Q-tables work great for small state spaces (grid worlds, tic-tac-toe) but fail for:
1. Image Inputs (Atari Games)
1State space size:
2- Screen: 210 × 160 pixels × 3 colors
3- Each pixel: 256 possible values
4- Total states: 256^(210×160×3) ≈ 10^127,000
5
6Q-table size: IMPOSSIBLE TO STORE
7(More states than atoms in the universe!)2. Continuous State Spaces (Robotics)
1Robot arm with 7 joints:
2- Each joint angle: continuous real value
3- State space: ℝ^7 (infinite states)
4
5Q-table: Cannot discretize without losing precision3. Generalization Problem
1Tabular Q-Learning:
2- Learns Q(s,a) for each (state, action) independently
3- Seeing state s₁ teaches NOTHING about similar state s₂
4- No generalization across similar situationsReplace Q-table with function approximator:
1# TABULAR Q-LEARNING (small state spaces)
2Q = np.zeros((n_states, n_actions)) # Lookup table
3q_value = Q[state, action] # Table lookup
4
5# DEEP Q-LEARNING (large state spaces)
6Q_network = NeuralNetwork() # Function approximator
7q_values = Q_network(state) # Forward pass
8q_value = q_values[action] # Select action's Q-valueAdvantages of Neural Networks:
1. Value-Based Methods (learn Q-function)
1Input: State (s)
2 ↓
3 Neural Network (Q-network)
4 ↓
5Output: Q(s, a₁), Q(s, a₂), ..., Q(s, aₙ)
6 ↓
7Action: argmax Q(s, a)Examples: DQN, Double DQN, Dueling DQN
2. Policy-Based Methods (learn policy directly)
1Input: State (s)
2 ↓
3 Neural Network (policy network)
4 ↓
5Output: π(a₁|s), π(a₂|s), ..., π(aₙ|s)
6 ↓
7Action: Sample from π(·|s)Examples: REINFORCE, PPO, TRPO
3. Actor-Critic Methods (learn both)
1Input: State (s)
2 ↓
3 ┌──────────────────┐
4 │ Actor Network │ → Policy π(a|s)
5 │ (Policy) │
6 └──────────────────┘
7 ↓
8 ┌──────────────────┐
9 │ Critic Network │ → Value V(s)
10 │ (Value) │
11 └──────────────────┘Examples: A3C, SAC, TD3
| Method | Pros | Cons | Best For |
|--------|------|------|----------|
| Value-Based (DQN) | Sample efficient, stable | Only discrete actions | Atari games, discrete control |
| Policy-Based (REINFORCE) | Continuous actions, stochastic policies | High variance | Continuous control, exploration |
| Actor-Critic (PPO) | Best of both worlds | More complex | General purpose, production |
1. Non-Stationarity
2. Sample Correlation
3. Deadly Triad (Sutton & Barto)
4. Credit Assignment
Deep Q-Network (DQN) combines Q-Learning with deep neural networks. Published by DeepMind in 2015 (Nature paper), DQN achieved human-level performance on 49 Atari games.
1Input: 84×84×4 stacked frames (grayscale Atari screens)
2 ↓
3 Conv2D(32, 8×8, stride=4) + ReLU
4 ↓
5 Conv2D(64, 4×4, stride=2) + ReLU
6 ↓
7 Conv2D(64, 3×3, stride=1) + ReLU
8 ↓
9 Flatten → Dense(512) + ReLU
10 ↓
11 Dense(n_actions) ← Q-values for each action
12 ↓
13Output: [Q(s, a₁), Q(s, a₂), ..., Q(s, aₙ)]1. Experience Replay
Problem: Sequential data is correlated → unstable learning
1# Store transitions in replay buffer
2replay_buffer = []
3for step in range(max_steps):
4 action = select_action(state)
5 next_state, reward, done = env.step(action)
6
7 # Store transition
8 replay_buffer.append((state, action, reward, next_state, done))
9
10 # Sample RANDOM minibatch for training
11 batch = random.sample(replay_buffer, batch_size)
12
13 # Train on batch (breaks correlation!)
14 train_on_batch(batch)Benefits:
2. Target Network
Problem: Q-network changes → target changes → chasing moving target
1# TWO networks: Q-network and Target network
2
3Q_network = NeuralNetwork() # Updated every step
4Q_target = copy(Q_network) # Updated every C steps
5
6# Loss computation
7for (s, a, r, s', done) in batch:
8 # Compute target using TARGET network (stable)
9 if done:
10 target = r
11 else:
12 target = r + gamma * max(Q_target(s'))
13
14 # Compute prediction using Q-network (learning)
15 prediction = Q_network(s)[a]
16
17 # MSE loss
18 loss = (target - prediction)^2
19
20# Update Q-network every step
21Q_network.update(loss)
22
23# Update target network every C=10,000 steps
24if step % C == 0:
25 Q_target = copy(Q_network)Benefits:
3. Frame Stacking
Problem: Single frame lacks temporal information (velocity, direction)
1# Stack last 4 frames as input
2state = stack([frame_t, frame_t-1, frame_t-2, frame_t-3])
3
4# Network can infer:
5# - Velocity (position change between frames)
6# - Direction (trajectory across frames)
7# - Acceleration (velocity change)1import torch
2import torch.nn as nn
3import torch.optim as optim
4import numpy as np
5from collections import deque
6import random
7
8class DQN(nn.Module):
9 """
10 Deep Q-Network for Atari games
11 """
12
13 def __init__(self, n_actions):
14 super(DQN, self).__init__()
15
16 # Convolutional layers
17 self.conv = nn.Sequential(
18 nn.Conv2d(4, 32, kernel_size=8, stride=4),
19 nn.ReLU(),
20 nn.Conv2d(32, 64, kernel_size=4, stride=2),
21 nn.ReLU(),
22 nn.Conv2d(64, 64, kernel_size=3, stride=1),
23 nn.ReLU()
24 )
25
26 # Fully connected layers
27 self.fc = nn.Sequential(
28 nn.Linear(7 * 7 * 64, 512),
29 nn.ReLU(),
30 nn.Linear(512, n_actions)
31 )
32
33 def forward(self, x):
34 """Forward pass: state → Q-values"""
35 x = self.conv(x)
36 x = x.view(x.size(0), -1) # Flatten
37 q_values = self.fc(x)
38 return q_values
39
40
41class ReplayBuffer:
42 """
43 Experience Replay Buffer
44 """
45
46 def __init__(self, capacity=100000):
47 self.buffer = deque(maxlen=capacity)
48
49 def push(self, state, action, reward, next_state, done):
50 """Store transition"""
51 self.buffer.append((state, action, reward, next_state, done))
52
53 def sample(self, batch_size):
54 """Sample random minibatch"""
55 return random.sample(self.buffer, batch_size)
56
57 def __len__(self):
58 return len(self.buffer)
59
60
61class DQNAgent:
62 """
63 DQN Agent with experience replay and target network
64 """
65
66 def __init__(self, n_actions, learning_rate=1e-4,
67 gamma=0.99, epsilon=1.0, epsilon_decay=0.9999,
68 epsilon_min=0.1, buffer_size=100000,
69 batch_size=32, target_update=10000):
70
71 self.n_actions = n_actions
72 self.gamma = gamma
73 self.epsilon = epsilon
74 self.epsilon_decay = epsilon_decay
75 self.epsilon_min = epsilon_min
76 self.batch_size = batch_size
77 self.target_update = target_update
78
79 # Q-network and target network
80 self.q_network = DQN(n_actions)
81 self.target_network = DQN(n_actions)
82 self.target_network.load_state_dict(self.q_network.state_dict())
83
84 # Optimizer and loss
85 self.optimizer = optim.Adam(self.q_network.parameters(),
86 lr=learning_rate)
87 self.loss_fn = nn.MSELoss()
88
89 # Experience replay
90 self.replay_buffer = ReplayBuffer(buffer_size)
91
92 # Training step counter
93 self.steps = 0
94
95 def select_action(self, state, training=True):
96 """
97 ε-greedy action selection
98
99 state: (4, 84, 84) numpy array
100 """
101 if training and random.random() < self.epsilon:
102 # Explore
103 return random.randint(0, self.n_actions - 1)
104 else:
105 # Exploit
106 with torch.no_grad():
107 state_tensor = torch.FloatTensor(state).unsqueeze(0)
108 q_values = self.q_network(state_tensor)
109 return q_values.argmax(1).item()
110
111 def train_step(self):
112 """
113 Single training step: sample batch and update Q-network
114 """
115 if len(self.replay_buffer) < self.batch_size:
116 return None
117
118 # Sample minibatch
119 batch = self.replay_buffer.sample(self.batch_size)
120 states, actions, rewards, next_states, dones = zip(*batch)
121
122 # Convert to tensors
123 states = torch.FloatTensor(np.array(states))
124 actions = torch.LongTensor(actions)
125 rewards = torch.FloatTensor(rewards)
126 next_states = torch.FloatTensor(np.array(next_states))
127 dones = torch.FloatTensor(dones)
128
129 # Current Q-values
130 current_q = self.q_network(states).gather(1, actions.unsqueeze(1))
131
132 # Target Q-values (using target network)
133 with torch.no_grad():
134 next_q = self.target_network(next_states).max(1)[0]
135 target_q = rewards + (1 - dones) * self.gamma * next_q
136
137 # Compute loss
138 loss = self.loss_fn(current_q.squeeze(), target_q)
139
140 # Optimize
141 self.optimizer.zero_grad()
142 loss.backward()
143 self.optimizer.step()
144
145 # Decay epsilon
146 self.epsilon = max(self.epsilon_min,
147 self.epsilon * self.epsilon_decay)
148
149 # Update target network
150 self.steps += 1
151 if self.steps % self.target_update == 0:
152 self.target_network.load_state_dict(
153 self.q_network.state_dict()
154 )
155
156 return loss.item()
157
158
159# Training loop
160def train_dqn(env, agent, n_episodes=1000):
161 """Train DQN agent"""
162
163 for episode in range(n_episodes):
164 state = env.reset()
165 episode_reward = 0
166 done = False
167
168 while not done:
169 # Select action
170 action = agent.select_action(state)
171
172 # Take action
173 next_state, reward, done, _ = env.step(action)
174
175 # Store transition
176 agent.replay_buffer.push(state, action, reward,
177 next_state, done)
178
179 # Train
180 loss = agent.train_step()
181
182 state = next_state
183 episode_reward += reward
184
185 if (episode + 1) % 10 == 0:
186 print(f"Episode {episode+1}: "
187 f"Reward={episode_reward:.2f}, "
188 f"ε={agent.epsilon:.3f}")
189
190 return agentDouble DQN (DDQN) - Reduces overestimation bias
1# Standard DQN
2target = r + γ max_a' Q_target(s', a')
3
4# Double DQN: use Q-network to SELECT action,
5# target network to EVALUATE it
6best_action = argmax_a' Q_network(s', a')
7target = r + γ Q_target(s', best_action)Dueling DQN - Separate value and advantage streams
1# Split network into:
2V(s): state value
3A(s,a): advantage of each action
4
5Q(s,a) = V(s) + (A(s,a) - mean(A(s,·)))Prioritized Experience Replay - Sample important transitions more
1# Assign priority based on TD error
2priority = |target - Q(s,a)|
3
4# Sample with probability ∝ priority^αBest For:
Not Suitable For:
Policy Gradient (PG) methods optimize the policy π_θ(a|s) directly, instead of learning Q-values.
Advantages over value-based methods:
1 DQN: argmax_a Q(s,a) → requires discrete actions
2 PG: π_θ(a|s) → can output continuous values1 Rock-Paper-Scissors: optimal policy is uniform random
2 Exploration: built into stochastic policyObjective: Maximize expected return
1J(θ) = 𝔼[Σ γ^t R_t | π_θ]Policy Gradient:
1∇_θ J(θ) = 𝔼[∇_θ log π_θ(a|s) · Q^π(s,a)]
2
3Intuition:
4- If Q(s,a) > 0: increase probability of action a
5- If Q(s,a) < 0: decrease probability of action a
6- Weighted by gradient of log-probabilityREINFORCE (Monte Carlo Policy Gradient) is the simplest PG algorithm.
1# Pseudocode
2Initialize policy network π_θ
3
4For each episode:
5 # Generate trajectory
6 τ = (s₀, a₀, r₁, s₁, a₁, r₂, ..., s_T)
7
8 # Compute returns
9 for t in range(T):
10 G_t = Σ_{k=t}^T γ^(k-t) r_k # Return from time t
11
12 # Update policy
13 for t in range(T):
14 θ = θ + α · ∇_θ log π_θ(a_t|s_t) · G_tComplete Implementation:
1import torch
2import torch.nn as nn
3import torch.optim as optim
4import numpy as np
5
6class PolicyNetwork(nn.Module):
7 """
8 Policy network for continuous action space
9 Outputs mean and std of Gaussian distribution
10 """
11
12 def __init__(self, state_dim, action_dim, hidden_dim=128):
13 super(PolicyNetwork, self).__init__()
14
15 self.fc = nn.Sequential(
16 nn.Linear(state_dim, hidden_dim),
17 nn.ReLU(),
18 nn.Linear(hidden_dim, hidden_dim),
19 nn.ReLU()
20 )
21
22 # Output mean and log_std
23 self.mean_layer = nn.Linear(hidden_dim, action_dim)
24 self.log_std_layer = nn.Linear(hidden_dim, action_dim)
25
26 def forward(self, state):
27 """Return distribution parameters"""
28 x = self.fc(state)
29 mean = self.mean_layer(x)
30 log_std = self.log_std_layer(x)
31 std = torch.exp(log_std)
32 return mean, std
33
34 def sample_action(self, state):
35 """Sample action from policy distribution"""
36 mean, std = self.forward(state)
37
38 # Sample from Gaussian
39 dist = torch.distributions.Normal(mean, std)
40 action = dist.sample()
41
42 # Log probability (for gradient)
43 log_prob = dist.log_prob(action).sum(-1)
44
45 return action, log_prob
46
47
48class REINFORCEAgent:
49 """
50 REINFORCE (Monte Carlo Policy Gradient) Agent
51 """
52
53 def __init__(self, state_dim, action_dim,
54 learning_rate=1e-3, gamma=0.99):
55 self.gamma = gamma
56
57 # Policy network
58 self.policy = PolicyNetwork(state_dim, action_dim)
59 self.optimizer = optim.Adam(self.policy.parameters(),
60 lr=learning_rate)
61
62 # Episode storage
63 self.saved_log_probs = []
64 self.rewards = []
65
66 def select_action(self, state):
67 """Select action and save log probability"""
68 state_tensor = torch.FloatTensor(state).unsqueeze(0)
69 action, log_prob = self.policy.sample_action(state_tensor)
70
71 self.saved_log_probs.append(log_prob)
72 return action.detach().numpy()[0]
73
74 def compute_returns(self):
75 """Compute discounted returns G_t"""
76 returns = []
77 G = 0
78 for r in reversed(self.rewards):
79 G = r + self.gamma * G
80 returns.insert(0, G)
81 return returns
82
83 def update(self):
84 """REINFORCE update at end of episode"""
85
86 # Compute returns
87 returns = self.compute_returns()
88 returns = torch.FloatTensor(returns)
89
90 # Normalize returns (reduce variance)
91 returns = (returns - returns.mean()) / (returns.std() + 1e-8)
92
93 # Policy gradient loss
94 policy_loss = []
95 for log_prob, G in zip(self.saved_log_probs, returns):
96 policy_loss.append(-log_prob * G) # Negative for gradient ascent
97
98 policy_loss = torch.stack(policy_loss).sum()
99
100 # Optimize
101 self.optimizer.zero_grad()
102 policy_loss.backward()
103 self.optimizer.step()
104
105 # Clear episode data
106 self.saved_log_probs = []
107 self.rewards = []
108
109 return policy_loss.item()
110
111
112# Training loop
113def train_reinforce(env, agent, n_episodes=1000):
114 """Train REINFORCE agent"""
115
116 for episode in range(n_episodes):
117 state = env.reset()
118 episode_reward = 0
119 done = False
120
121 # Generate episode
122 while not done:
123 action = agent.select_action(state)
124 next_state, reward, done, _ = env.step(action)
125
126 agent.rewards.append(reward)
127 episode_reward += reward
128 state = next_state
129
130 # Update policy at end of episode
131 loss = agent.update()
132
133 if (episode + 1) % 10 == 0:
134 print(f"Episode {episode+1}: "
135 f"Reward={episode_reward:.2f}, "
136 f"Loss={loss:.3f}")
137
138 return agentProblem: REINFORCE has high variance → slow learning
Solution: Subtract baseline b(s) from returns
1∇_θ J(θ) = 𝔼[∇_θ log π_θ(a|s) · (G_t - b(s))]
2
3Common baselines:
4- Constant: b(s) = mean return
5- State-dependent: b(s) = V(s) ← Actor-Critic!With baseline:
This leads to Actor-Critic methods!
Two networks:
1class ActorCritic(nn.Module):
2 """
3 Actor-Critic with shared backbone
4 """
5
6 def __init__(self, state_dim, action_dim, hidden_dim=128):
7 super().__init__()
8
9 # Shared layers
10 self.shared = nn.Sequential(
11 nn.Linear(state_dim, hidden_dim),
12 nn.ReLU(),
13 nn.Linear(hidden_dim, hidden_dim),
14 nn.ReLU()
15 )
16
17 # Actor head (policy)
18 self.actor = nn.Linear(hidden_dim, action_dim)
19
20 # Critic head (value)
21 self.critic = nn.Linear(hidden_dim, 1)
22
23 def forward(self, state):
24 x = self.shared(state)
25 action_probs = torch.softmax(self.actor(x), dim=-1)
26 value = self.critic(x)
27 return action_probs, value
28
29
30# Actor-Critic update
31def actor_critic_update(agent, state, action, reward,
32 next_state, done):
33 """Single-step Actor-Critic (A2C)"""
34
35 # Get current policy and value
36 probs, value = agent(state)
37 action_prob = probs[action]
38
39 # Compute TD target and error
40 with torch.no_grad():
41 _, next_value = agent(next_state)
42 td_target = reward + (1 - done) * gamma * next_value
43 advantage = td_target - value # A(s,a) = Q(s,a) - V(s)
44
45 # Actor loss: policy gradient with advantage
46 actor_loss = -torch.log(action_prob) * advantage.detach()
47
48 # Critic loss: TD error
49 critic_loss = (td_target - value) ** 2
50
51 # Total loss
52 loss = actor_loss + critic_loss
53 loss.backward()Advantages:
Actor-Critic Variants:
Proximal Policy Optimization (PPO) is the state-of-the-art policy gradient algorithm in October 2025. It's used in production at OpenAI, DeepMind, and across industry.
Problems with vanilla policy gradients:
PPO Solutions:
Standard policy gradient:
1L(θ) = 𝔼[log π_θ(a|s) · A(s,a)]PPO clipped objective:
1r(θ) = π_θ(a|s) / π_θ_old(a|s) ← probability ratio
2
3L_CLIP(θ) = 𝔼[min(
4 r(θ) · A(s,a),
5 clip(r(θ), 1-ε, 1+ε) · A(s,a)
6)]
7
8where ε = 0.2 (typical value)Intuition:
Visualization:
1Advantage > 0:
2 Objective
3 │ ╱───── ← clipped (can't increase more)
4 │ ╱
5 │ ╱
6 └────────── r(θ)
7 1 1+ε
8
9Advantage < 0:
10 Objective
11 │
12 │ ╲
13 │ ╲
14 └─────╲──── ← clipped
15 1-ε 1 r(θ)1import torch
2import torch.nn as nn
3import torch.optim as optim
4import numpy as np
5
6class PPOMemory:
7 """
8 Storage for PPO training data
9 """
10
11 def __init__(self):
12 self.states = []
13 self.actions = []
14 self.log_probs = []
15 self.rewards = []
16 self.values = []
17 self.dones = []
18
19 def store(self, state, action, log_prob, reward, value, done):
20 self.states.append(state)
21 self.actions.append(action)
22 self.log_probs.append(log_prob)
23 self.rewards.append(reward)
24 self.values.append(value)
25 self.dones.append(done)
26
27 def clear(self):
28 self.states = []
29 self.actions = []
30 self.log_probs = []
31 self.rewards = []
32 self.values = []
33 self.dones = []
34
35 def get_batches(self):
36 """Return data as tensors"""
37 return (
38 torch.FloatTensor(np.array(self.states)),
39 torch.LongTensor(self.actions),
40 torch.FloatTensor(self.log_probs),
41 torch.FloatTensor(self.rewards),
42 torch.FloatTensor(self.values),
43 torch.FloatTensor(self.dones)
44 )
45
46
47class ActorCriticPPO(nn.Module):
48 """
49 Actor-Critic network for PPO
50 """
51
52 def __init__(self, state_dim, action_dim, hidden_dim=64):
53 super().__init__()
54
55 # Shared backbone
56 self.backbone = nn.Sequential(
57 nn.Linear(state_dim, hidden_dim),
58 nn.Tanh(),
59 nn.Linear(hidden_dim, hidden_dim),
60 nn.Tanh()
61 )
62
63 # Actor (policy)
64 self.actor = nn.Linear(hidden_dim, action_dim)
65
66 # Critic (value)
67 self.critic = nn.Linear(hidden_dim, 1)
68
69 def forward(self, state):
70 x = self.backbone(state)
71 action_logits = self.actor(x)
72 value = self.critic(x)
73 return action_logits, value
74
75 def get_action(self, state):
76 """Sample action and return log probability"""
77 action_logits, value = self.forward(state)
78
79 # Create categorical distribution
80 dist = torch.distributions.Categorical(logits=action_logits)
81 action = dist.sample()
82 log_prob = dist.log_prob(action)
83
84 return action.item(), log_prob.item(), value.item()
85
86 def evaluate_actions(self, states, actions):
87 """
88 Evaluate actions for PPO update
89
90 Returns:
91 - log_probs: log π_θ(a|s)
92 - values: V(s)
93 - entropy: policy entropy (for exploration bonus)
94 """
95 action_logits, values = self.forward(states)
96
97 dist = torch.distributions.Categorical(logits=action_logits)
98 log_probs = dist.log_prob(actions)
99 entropy = dist.entropy()
100
101 return log_probs, values.squeeze(), entropy
102
103
104class PPOAgent:
105 """
106 Proximal Policy Optimization Agent
107 """
108
109 def __init__(self, state_dim, action_dim,
110 learning_rate=3e-4, gamma=0.99, gae_lambda=0.95,
111 clip_epsilon=0.2, value_coef=0.5, entropy_coef=0.01,
112 n_epochs=10, batch_size=64):
113
114 self.gamma = gamma
115 self.gae_lambda = gae_lambda
116 self.clip_epsilon = clip_epsilon
117 self.value_coef = value_coef
118 self.entropy_coef = entropy_coef
119 self.n_epochs = n_epochs
120 self.batch_size = batch_size
121
122 # Actor-Critic network
123 self.ac = ActorCriticPPO(state_dim, action_dim)
124 self.optimizer = optim.Adam(self.ac.parameters(), lr=learning_rate)
125
126 # Memory
127 self.memory = PPOMemory()
128
129 def select_action(self, state):
130 """Select action during data collection"""
131 state_tensor = torch.FloatTensor(state).unsqueeze(0)
132
133 with torch.no_grad():
134 action, log_prob, value = self.ac.get_action(state_tensor)
135
136 return action, log_prob, value
137
138 def compute_gae(self, rewards, values, dones):
139 """
140 Compute Generalized Advantage Estimation (GAE)
141
142 GAE balances bias-variance tradeoff:
143 - λ=0: A(s,a) = δ_t (low variance, high bias)
144 - λ=1: A(s,a) = Σ γ^k δ_{t+k} (high variance, low bias)
145 """
146 advantages = []
147 gae = 0
148
149 # Reverse iteration
150 for t in reversed(range(len(rewards))):
151 if t == len(rewards) - 1:
152 next_value = 0
153 else:
154 next_value = values[t + 1]
155
156 # TD error
157 delta = rewards[t] + self.gamma * next_value * (1 - dones[t]) - values[t]
158
159 # GAE
160 gae = delta + self.gamma * self.gae_lambda * (1 - dones[t]) * gae
161 advantages.insert(0, gae)
162
163 return torch.FloatTensor(advantages)
164
165 def update(self):
166 """PPO update using collected data"""
167
168 # Get data
169 states, actions, old_log_probs, rewards, values, dones = self.memory.get_batches()
170
171 # Compute advantages
172 advantages = self.compute_gae(rewards, values, dones)
173 advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
174
175 # Compute returns
176 returns = advantages + values
177
178 # Multiple epochs of optimization
179 for _ in range(self.n_epochs):
180 # Mini-batch SGD
181 indices = np.arange(len(states))
182 np.random.shuffle(indices)
183
184 for start in range(0, len(states), self.batch_size):
185 end = start + self.batch_size
186 batch_indices = indices[start:end]
187
188 batch_states = states[batch_indices]
189 batch_actions = actions[batch_indices]
190 batch_old_log_probs = old_log_probs[batch_indices]
191 batch_advantages = advantages[batch_indices]
192 batch_returns = returns[batch_indices]
193
194 # Evaluate actions with current policy
195 log_probs, values, entropy = self.ac.evaluate_actions(
196 batch_states, batch_actions
197 )
198
199 # Importance sampling ratio
200 ratio = torch.exp(log_probs - batch_old_log_probs)
201
202 # Clipped surrogate objective
203 surr1 = ratio * batch_advantages
204 surr2 = torch.clamp(ratio,
205 1 - self.clip_epsilon,
206 1 + self.clip_epsilon) * batch_advantages
207 actor_loss = -torch.min(surr1, surr2).mean()
208
209 # Value loss
210 value_loss = ((batch_returns - values) ** 2).mean()
211
212 # Entropy bonus (encourages exploration)
213 entropy_loss = -entropy.mean()
214
215 # Total loss
216 loss = (actor_loss +
217 self.value_coef * value_loss +
218 self.entropy_coef * entropy_loss)
219
220 # Optimize
221 self.optimizer.zero_grad()
222 loss.backward()
223 torch.nn.utils.clip_grad_norm_(self.ac.parameters(), 0.5)
224 self.optimizer.step()
225
226 # Clear memory
227 self.memory.clear()
228
229
230# Training loop
231def train_ppo(env, agent, n_episodes=1000, horizon=2048):
232 """Train PPO agent"""
233
234 state = env.reset()
235 episode_reward = 0
236 episode_count = 0
237
238 for step in range(n_episodes * horizon):
239 # Collect data
240 action, log_prob, value = agent.select_action(state)
241 next_state, reward, done, _ = env.step(action)
242
243 agent.memory.store(state, action, log_prob, reward, value, done)
244 episode_reward += reward
245
246 if done:
247 state = env.reset()
248 episode_count += 1
249 print(f"Episode {episode_count}: Reward = {episode_reward:.2f}")
250 episode_reward = 0
251 else:
252 state = next_state
253
254 # Update policy every horizon steps
255 if (step + 1) % horizon == 0:
256 agent.update()
257
258 return agentApplications:
Why PPO is Popular:
Hyperparameters (robust defaults):
1learning_rate = 3e-4
2gamma = 0.99
3gae_lambda = 0.95
4clip_epsilon = 0.2
5n_epochs = 10
6batch_size = 64
7horizon = 2048These work well across most tasks with minimal tuning!
Deep RL continues to evolve rapidly. Here are cutting-edge techniques and future directions as of October 2025.
Idea: Learn environment model, use it for planning
1Model-Free (DQN, PPO):
2 state → policy → action → reward
3 (learn policy from experience)
4
5Model-Based:
6 1. Learn dynamics: (s, a) → s'
7 2. Learn reward: (s, a) → r
8 3. Plan using model: simulate trajectories
9
10Advantages:
11 ✓ Sample efficient (reuse data via simulation)
12 ✓ Can plan ahead (search in model)
13 ✗ Model errors compound (inaccurate predictions)Algorithms:
Problem: Online RL needs environment interaction (dangerous/expensive)
Offline RL: Learn from fixed dataset (no interaction)
1# Offline RL workflow
2dataset = load_dataset() # Pre-collected transitions
3
4for batch in dataset:
5 policy.update(batch) # Learn from static data
6
7# Deploy learned policy
8# No exploration during training!Challenges:
Algorithms:
Use Cases:
Multiple agents learning simultaneously
1Challenges:
2- Non-stationarity (other agents' policies change)
3- Credit assignment (who deserves reward?)
4- Coordination vs competition
5
6Approaches:
7- Independent learners (each agent learns separately)
8- Centralized training, decentralized execution (CTDE)
9- Communication between agentsAlgorithms:
Applications:
Learn policies at multiple temporal scales
1High-level policy (manager):
2 "Go to kitchen" → "Open fridge" → "Grab milk"
3
4Low-level policy (worker):
5 "Open fridge" → (move arm, grasp handle, pull)
6
7Benefits:
8 ✓ Exploration efficiency (reuse skills)
9 ✓ Transfer learning (share low-level policies)
10 ✓ Temporal abstraction (long-horizon planning)Algorithms:
Train agent to quickly adapt to new tasks
1# Meta-training (learn how to learn)
2for task in task_distribution:
3 # Few-shot adaptation
4 adapt_policy(task, few_examples)
5 test_performance(task)
6
7# Meta-policy learns to:
8# - Extract task information quickly
9# - Adapt policy efficientlyAlgorithms:
Applications:
The technique behind ChatGPT and Claude
11. Supervised fine-tuning
2 → Train LLM on high-quality demonstrations
3
42. Reward modeling
5 → Humans rank outputs: A > B
6 → Train reward model to predict preferences
7
83. RL optimization (PPO)
9 → Use reward model as environment
10 → Optimize LLM to maximize predicted reward
11 → Add KL penalty to stay close to supervised modelWhy it works:
Current Research (October 2025):
Critical for real-world deployment
Safe RL:
Robust RL:
You've completed the Deep RL lesson! You now understand:
Next: In Lesson 20: MLOps & Model Deployment, you'll learn how to take your AI/ML models from notebook to production—covering experiment tracking, model serving, Docker containerization, and production monitoring.
The final lesson completes your journey from AI fundamentals to production deployment! 🚀
Implement Proximal Policy Optimization (PPO) to solve the CartPole continuous control task. The agent learns to balance a pole on a moving cart using policy gradient methods with clipped objectives.
1# Complete PPO Implementation for CartPole
2# State-of-the-art deep reinforcement learning
3
4import torch
5import torch.nn as nn
6import torch.optim as optim
7import numpy as np
8import matplotlib.pyplot as plt
9from collections import deque
10
11print("=" * 70)
12print("PROXIMAL POLICY OPTIMIZATION (PPO)")
13print("=" * 70)
14print()
15print("Building state-of-the-art RL agent for CartPole-v1")
16print("Algorithm: PPO with GAE (Generalized Advantage Estimation)")
17print()
18
19# ============================================================================
20# PART 1: Environment Wrapper
21# ============================================================================
22
23class CartPoleEnv:
24 """
25 Simplified CartPole environment
26
27 State: [cart_position, cart_velocity, pole_angle, pole_angular_velocity]
28 Action: 0 (left) or 1 (right)
29 Goal: Keep pole balanced (angle < 12°) for 500 steps
30 """
31
32 def __init__(self):
33 self.gravity = 9.8
34 self.masscart = 1.0
35 self.masspole = 0.1
36 self.total_mass = self.masspole + self.masscart
37 self.length = 0.5
38 self.polemass_length = self.masspole * self.length
39 self.force_mag = 10.0
40 self.tau = 0.02 # seconds between state updates
41
42 # Angle at which to fail episode
43 self.theta_threshold_radians = 12 * 2 * np.pi / 360
44 self.x_threshold = 2.4
45
46 self.state = None
47 self.steps = 0
48
49 def reset(self):
50 """Reset environment to initial state"""
51 self.state = np.random.uniform(low=-0.05, high=0.05, size=(4,))
52 self.steps = 0
53 return self.state.copy()
54
55 def step(self, action):
56 """
57 Take action, return (next_state, reward, done, info)
58 """
59 x, x_dot, theta, theta_dot = self.state
60 force = self.force_mag if action == 1 else -self.force_mag
61
62 # Physics simulation
63 costheta = np.cos(theta)
64 sintheta = np.sin(theta)
65
66 temp = (force + self.polemass_length * theta_dot ** 2 * sintheta) / self.total_mass
67 thetaacc = (self.gravity * sintheta - costheta * temp) / (self.length * (4.0 / 3.0 - self.masspole * costheta ** 2 / self.total_mass))
68 xacc = temp - self.polemass_length * thetaacc * costheta / self.total_mass
69
70 # Update state
71 x = x + self.tau * x_dot
72 x_dot = x_dot + self.tau * xacc
73 theta = theta + self.tau * theta_dot
74 theta_dot = theta_dot + self.tau * thetaacc
75
76 self.state = np.array([x, x_dot, theta, theta_dot])
77 self.steps += 1
78
79 # Check termination
80 done = bool(
81 x < -self.x_threshold
82 or x > self.x_threshold
83 or theta < -self.theta_threshold_radians
84 or theta > self.theta_threshold_radians
85 or self.steps >= 500
86 )
87
88 # Reward: +1 for each step survived
89 reward = 1.0 if not done else 0.0
90
91 return self.state.copy(), reward, done, {}
92
93
94# ============================================================================
95# PART 2: PPO Networks
96# ============================================================================
97
98class ActorCriticNetwork(nn.Module):
99 """
100 Actor-Critic network for PPO
101
102 Actor: π_θ(a|s) outputs action probabilities
103 Critic: V_ϕ(s) outputs state value estimate
104 """
105
106 def __init__(self, state_dim=4, action_dim=2, hidden_dim=64):
107 super(ActorCriticNetwork, self).__init__()
108
109 # Shared feature extractor
110 self.shared = nn.Sequential(
111 nn.Linear(state_dim, hidden_dim),
112 nn.Tanh(),
113 nn.Linear(hidden_dim, hidden_dim),
114 nn.Tanh()
115 )
116
117 # Actor head (policy network)
118 self.actor = nn.Linear(hidden_dim, action_dim)
119
120 # Critic head (value network)
121 self.critic = nn.Linear(hidden_dim, 1)
122
123 def forward(self, state):
124 """
125 Forward pass
126
127 Returns:
128 - action_logits: unnormalized log probabilities
129 - value: state value V(s)
130 """
131 features = self.shared(state)
132 action_logits = self.actor(features)
133 value = self.critic(features)
134 return action_logits, value.squeeze()
135
136 def get_action(self, state):
137 """
138 Sample action from policy
139
140 Returns:
141 - action: sampled action
142 - log_prob: log π_θ(a|s)
143 - value: V(s)
144 """
145 action_logits, value = self.forward(state)
146
147 # Create categorical distribution
148 dist = torch.distributions.Categorical(logits=action_logits)
149
150 # Sample action
151 action = dist.sample()
152 log_prob = dist.log_prob(action)
153
154 return action.item(), log_prob.item(), value.item()
155
156 def evaluate_actions(self, states, actions):
157 """
158 Evaluate actions for PPO update
159
160 Returns:
161 - log_probs: log π_θ(a|s) for each action
162 - values: V(s) for each state
163 - entropy: policy entropy H(π(·|s))
164 """
165 action_logits, values = self.forward(states)
166
167 dist = torch.distributions.Categorical(logits=action_logits)
168 log_probs = dist.log_prob(actions)
169 entropy = dist.entropy().mean() # Entropy bonus for exploration
170
171 return log_probs, values, entropy
172
173
174# ============================================================================
175# PART 3: PPO Memory Buffer
176# ============================================================================
177
178class PPOMemory:
179 """
180 Rollout buffer for PPO training data
181 """
182
183 def __init__(self):
184 self.states = []
185 self.actions = []
186 self.log_probs = []
187 self.rewards = []
188 self.values = []
189 self.dones = []
190
191 def store(self, state, action, log_prob, reward, value, done):
192 """Store transition"""
193 self.states.append(state)
194 self.actions.append(action)
195 self.log_probs.append(log_prob)
196 self.rewards.append(reward)
197 self.values.append(value)
198 self.dones.append(1 if done else 0)
199
200 def clear(self):
201 """Clear buffer"""
202 self.states = []
203 self.actions = []
204 self.log_probs = []
205 self.rewards = []
206 self.values = []
207 self.dones = []
208
209 def get_batches(self):
210 """Convert to tensors"""
211 return (
212 torch.FloatTensor(np.array(self.states)),
213 torch.LongTensor(self.actions),
214 torch.FloatTensor(self.log_probs),
215 torch.FloatTensor(self.rewards),
216 torch.FloatTensor(self.values),
217 torch.FloatTensor(self.dones)
218 )
219
220 def __len__(self):
221 return len(self.states)
222
223
224# ============================================================================
225# PART 4: PPO Agent
226# ============================================================================
227
228class PPOAgent:
229 """
230 Proximal Policy Optimization Agent
231 """
232
233 def __init__(self, state_dim=4, action_dim=2,
234 lr=3e-4, gamma=0.99, gae_lambda=0.95,
235 clip_epsilon=0.2, value_coef=0.5, entropy_coef=0.01,
236 n_epochs=10, batch_size=64):
237
238 self.gamma = gamma
239 self.gae_lambda = gae_lambda
240 self.clip_epsilon = clip_epsilon
241 self.value_coef = value_coef
242 self.entropy_coef = entropy_coef
243 self.n_epochs = n_epochs
244 self.batch_size = batch_size
245
246 # Actor-Critic network
247 self.network = ActorCriticNetwork(state_dim, action_dim)
248 self.optimizer = optim.Adam(self.network.parameters(), lr=lr)
249
250 # Memory buffer
251 self.memory = PPOMemory()
252
253 # Training stats
254 self.training_stats = {
255 'actor_losses': [],
256 'critic_losses': [],
257 'entropies': [],
258 'clip_fractions': []
259 }
260
261 def select_action(self, state):
262 """Select action using current policy"""
263 state_tensor = torch.FloatTensor(state).unsqueeze(0)
264
265 with torch.no_grad():
266 action, log_prob, value = self.network.get_action(state_tensor)
267
268 return action, log_prob, value
269
270 def compute_gae(self, rewards, values, dones):
271 """
272 Compute Generalized Advantage Estimation (GAE)
273
274 GAE(λ) = Σ (γλ)^t δ_t
275
276 where δ_t = r_t + γV(s_{t+1}) - V(s_t)
277
278 Benefits:
279 - Reduces variance compared to Monte Carlo
280 - Less biased than TD(0)
281 - Balances bias-variance with λ
282 """
283 advantages = []
284 gae = 0
285
286 # Backward iteration
287 for t in reversed(range(len(rewards))):
288 if t == len(rewards) - 1:
289 next_value = 0
290 else:
291 next_value = values[t + 1]
292
293 # TD error: δ_t = r_t + γV(s_{t+1}) - V(s_t)
294 delta = rewards[t] + self.gamma * next_value * (1 - dones[t]) - values[t]
295
296 # GAE: A_t = δ_t + (γλ)δ_{t+1} + (γλ)²δ_{t+2} + ...
297 gae = delta + self.gamma * self.gae_lambda * (1 - dones[t]) * gae
298 advantages.insert(0, gae)
299
300 return torch.FloatTensor(advantages)
301
302 def update(self):
303 """
304 PPO update using collected rollout data
305 """
306 # Get rollout data
307 states, actions, old_log_probs, rewards, values, dones = self.memory.get_batches()
308
309 # Compute advantages using GAE
310 advantages = self.compute_gae(rewards, values, dones)
311
312 # Normalize advantages (reduces variance)
313 advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
314
315 # Compute returns: R_t = A_t + V(s_t)
316 returns = advantages + values
317
318 # Multiple epochs of optimization
319 for epoch in range(self.n_epochs):
320 # Mini-batch SGD
321 indices = np.arange(len(states))
322 np.random.shuffle(indices)
323
324 for start in range(0, len(states), self.batch_size):
325 end = min(start + self.batch_size, len(states))
326 batch_idx = indices[start:end]
327
328 batch_states = states[batch_idx]
329 batch_actions = actions[batch_idx]
330 batch_old_log_probs = old_log_probs[batch_idx]
331 batch_advantages = advantages[batch_idx]
332 batch_returns = returns[batch_idx]
333
334 # Evaluate actions with current policy
335 log_probs, current_values, entropy = self.network.evaluate_actions(batch_states, batch_actions)
336
337 # Importance sampling ratio: π_θ(a|s) / π_θ_old(a|s)
338 ratio = torch.exp(log_probs - batch_old_log_probs)
339
340 # PPO clipped objective
341 surr1 = ratio * batch_advantages
342 surr2 = torch.clamp(ratio,
343 1 - self.clip_epsilon,
344 1 + self.clip_epsilon) * batch_advantages
345 actor_loss = -torch.min(surr1, surr2).mean()
346
347 # Value loss (MSE)
348 critic_loss = ((batch_returns - current_values) ** 2).mean()
349
350 # Entropy bonus (encourages exploration)
351 entropy_loss = -entropy
352
353 # Total loss
354 loss = (actor_loss +
355 self.value_coef * critic_loss +
356 self.entropy_coef * entropy_loss)
357
358 # Optimize
359 self.optimizer.zero_grad()
360 loss.backward()
361 torch.nn.utils.clip_grad_norm_(self.network.parameters(), 0.5)
362 self.optimizer.step()
363
364 # Track stats
365 with torch.no_grad():
366 clip_fraction = ((ratio < 1 - self.clip_epsilon) |
367 (ratio > 1 + self.clip_epsilon)).float().mean()
368
369 self.training_stats['actor_losses'].append(actor_loss.item())
370 self.training_stats['critic_losses'].append(critic_loss.item())
371 self.training_stats['entropies'].append(entropy.item())
372 self.training_stats['clip_fractions'].append(clip_fraction.item())
373
374 # Clear memory after update
375 self.memory.clear()
376
377
378# ============================================================================
379# PART 5: Training Loop
380# ============================================================================
381
382def train_ppo(n_episodes=500, horizon=2048, verbose=True):
383 """
384 Train PPO agent on CartPole
385 """
386 print("Initializing environment and agent...")
387 env = CartPoleEnv()
388 agent = PPOAgent(
389 state_dim=4,
390 action_dim=2,
391 lr=3e-4,
392 gamma=0.99,
393 gae_lambda=0.95,
394 clip_epsilon=0.2,
395 n_epochs=10,
396 batch_size=64
397 )
398
399 print("Training started...")
400 print()
401
402 episode_rewards = []
403 episode_lengths = []
404 current_episode_reward = 0
405 current_episode_length = 0
406
407 state = env.reset()
408 total_steps = 0
409
410 for step in range(n_episodes * horizon):
411 # Select action
412 action, log_prob, value = agent.select_action(state)
413
414 # Take action
415 next_state, reward, done, _ = env.step(action)
416
417 # Store transition
418 agent.memory.store(state, action, log_prob, reward, value, done)
419
420 current_episode_reward += reward
421 current_episode_length += 1
422 total_steps += 1
423
424 if done:
425 # Episode ended
426 episode_rewards.append(current_episode_reward)
427 episode_lengths.append(current_episode_length)
428
429 if verbose and len(episode_rewards) % 10 == 0:
430 avg_reward = np.mean(episode_rewards[-10:])
431 avg_length = np.mean(episode_lengths[-10:])
432 print(f"Episode {len(episode_rewards):3d} | "
433 f"Avg Reward: {avg_reward:6.1f} | "
434 f"Avg Length: {avg_length:5.1f}")
435
436 # Reset
437 state = env.reset()
438 current_episode_reward = 0
439 current_episode_length = 0
440 else:
441 state = next_state
442
443 # PPO update every horizon steps
444 if (step + 1) % horizon == 0:
445 agent.update()
446 if verbose:
447 print(f" → PPO update at step {step+1}")
448
449 # Stop if solved
450 if len(episode_rewards) >= 100:
451 if np.mean(episode_rewards[-100:]) >= 475:
452 print()
453 print(f"✅ Solved in {len(episode_rewards)} episodes!")
454 print(f" Average reward over last 100 episodes: "
455 f"{np.mean(episode_rewards[-100:]):.1f}")
456 break
457
458 return agent, episode_rewards, episode_lengths
459
460
461# ============================================================================
462# PART 6: Visualization
463# ============================================================================
464
465def visualize_training(episode_rewards, episode_lengths, agent):
466 """
467 Plot training progress and learned policy
468 """
469 fig = plt.figure(figsize=(16, 10))
470 gs = fig.add_gridspec(3, 2, hspace=0.3, wspace=0.3)
471
472 # Episode rewards
473 ax1 = fig.add_subplot(gs[0, 0])
474 ax1.plot(episode_rewards, alpha=0.3, color='blue')
475 window = 20
476 if len(episode_rewards) >= window:
477 smoothed = np.convolve(episode_rewards,
478 np.ones(window)/window, mode='valid')
479 ax1.plot(range(window-1, len(episode_rewards)), smoothed,
480 linewidth=2, color='darkblue', label=f'{window}-episode average')
481 ax1.axhline(y=475, color='red', linestyle='--',
482 linewidth=2, label='Solved threshold')
483 ax1.set_xlabel('Episode', fontweight='bold')
484 ax1.set_ylabel('Total Reward', fontweight='bold')
485 ax1.set_title('Learning Curve: Episode Rewards', fontweight='bold', fontsize=12)
486 ax1.legend()
487 ax1.grid(True, alpha=0.3)
488
489 # Episode lengths
490 ax2 = fig.add_subplot(gs[0, 1])
491 ax2.plot(episode_lengths, alpha=0.3, color='green')
492 if len(episode_lengths) >= window:
493 smoothed = np.convolve(episode_lengths,
494 np.ones(window)/window, mode='valid')
495 ax2.plot(range(window-1, len(episode_lengths)), smoothed,
496 linewidth=2, color='darkgreen', label=f'{window}-episode average')
497 ax2.set_xlabel('Episode', fontweight='bold')
498 ax2.set_ylabel('Episode Length', fontweight='bold')
499 ax2.set_title('Episode Lengths', fontweight='bold', fontsize=12)
500 ax2.legend()
501 ax2.grid(True, alpha=0.3)
502
503 # Training stats
504 if len(agent.training_stats['actor_losses']) > 0:
505 # Actor loss
506 ax3 = fig.add_subplot(gs[1, 0])
507 ax3.plot(agent.training_stats['actor_losses'], alpha=0.5, color='purple')
508 ax3.set_xlabel('Update Step', fontweight='bold')
509 ax3.set_ylabel('Actor Loss', fontweight='bold')
510 ax3.set_title('PPO Actor Loss (Policy)', fontweight='bold', fontsize=12)
511 ax3.grid(True, alpha=0.3)
512
513 # Critic loss
514 ax4 = fig.add_subplot(gs[1, 1])
515 ax4.plot(agent.training_stats['critic_losses'], alpha=0.5, color='orange')
516 ax4.set_xlabel('Update Step', fontweight='bold')
517 ax4.set_ylabel('Critic Loss', fontweight='bold')
518 ax4.set_title('PPO Critic Loss (Value)', fontweight='bold', fontsize=12)
519 ax4.grid(True, alpha=0.3)
520
521 # Entropy
522 ax5 = fig.add_subplot(gs[2, 0])
523 ax5.plot(agent.training_stats['entropies'], alpha=0.5, color='brown')
524 ax5.set_xlabel('Update Step', fontweight='bold')
525 ax5.set_ylabel('Entropy', fontweight='bold')
526 ax5.set_title('Policy Entropy (Exploration)', fontweight='bold', fontsize=12)
527 ax5.grid(True, alpha=0.3)
528
529 # Clip fraction
530 ax6 = fig.add_subplot(gs[2, 1])
531 ax6.plot(agent.training_stats['clip_fractions'], alpha=0.5, color='teal')
532 ax6.set_xlabel('Update Step', fontweight='bold')
533 ax6.set_ylabel('Clip Fraction', fontweight='bold')
534 ax6.set_title('PPO Clipping Fraction', fontweight='bold', fontsize=12)
535 ax6.grid(True, alpha=0.3)
536
537 fig.suptitle('PPO Training Analysis', fontsize=16, fontweight='bold')
538 plt.show()
539
540
541# ============================================================================
542# PART 7: Run Training
543# ============================================================================
544
545print()
546agent, rewards, lengths = train_ppo(n_episodes=500, horizon=2048, verbose=True)
547
548print()
549print("=" * 70)
550print("TRAINING COMPLETE")
551print("=" * 70)
552print()
553
554# Final statistics
555if len(rewards) >= 100:
556 final_100 = rewards[-100:]
557 print(f"Final 100 episodes statistics:")
558 print(f" Mean reward: {np.mean(final_100):.1f}")
559 print(f" Std reward: {np.std(final_100):.1f}")
560 print(f" Min reward: {np.min(final_100):.1f}")
561 print(f" Max reward: {np.max(final_100):.1f}")
562 print()
563
564# Visualize
565print("Generating training visualizations...")
566visualize_training(rewards, lengths, agent)
567
568print()
569print("=" * 70)
570print("✅ PPO DEMO COMPLETE")
571print("=" * 70)
572print()
573print("Key Takeaways:")
574print(" • PPO uses clipped objective to prevent large policy updates")
575print(" • GAE balances bias-variance in advantage estimation")
576print(" • Multiple epochs reuse data for sample efficiency")
577print(" • Entropy bonus encourages exploration")
578print(" • PPO is robust and works across many RL tasks")
579print()This comprehensive PPO implementation demonstrates state-of-the-art deep reinforcement learning.
1. Complete PPO Pipeline
The implementation includes all components of production-quality PPO:
2. PPO Clipped Objective
The core innovation that makes PPO robust:
1L_CLIP = 𝔼[min(r(θ)·A, clip(r(θ), 1-ε, 1+ε)·A)]This prevents:
3. Generalized Advantage Estimation (GAE)
GAE computes advantages that balance:
4. Training Dynamics
Episodes 1-50: Random exploration
Episodes 51-200: Effective learning
Episodes 201+: Policy refinement
5. Why PPO Dominates (October 2025)
Sample Efficiency: Multiple epochs reuse collected data
Stability: Clipped objective prevents catastrophic updates
Robustness: Same hyperparameters work across diverse tasks
Scalability: Parallelizes well (multiple actors collecting data)
Generality: Works for discrete AND continuous actions
This CartPole example demonstrates principles that power:
Large Language Models (RLHF):
Robotics:
Game AI:
Autonomous Systems:
1. Trust Region Optimization
PPO implicitly maintains a trust region around the old policy—ensuring policy changes remain conservative.
2. Value Function Learning
The critic learns V(s) which serves as:
3. Entropy Regularization
Adding -H(π) to loss encourages exploration by preventing premature convergence to deterministic policy.
4. Gradient Clipping
Clipping gradients prevents exploding gradients in the neural network, adding another layer of stability.
To scale this to real-world tasks:
You now understand the algorithm behind modern RL applications! In Lesson 20, you'll learn how to deploy these models to production.
Exceptional work! You've mastered Deep Reinforcement Learning—from Deep Q-Networks to policy gradients to state-of-the-art PPO. You can now build agents that learn complex behaviors from high-dimensional inputs.
Next: In Lesson 20: MLOps & Model Deployment, you'll complete your AI/ML journey by learning how to deploy models to production. You'll master:
You'll learn the complete MLOps lifecycle—taking your AI models from Jupyter notebooks to production systems serving millions of users.
Your Complete Journey: AI Fundamentals → Deep Learning → Computer Vision → NLP → Transformers → Generative Models → Reinforcement Learning → Production Deployment
Ready to deploy your AI systems to production? Let's build production-grade MLOps pipelines! 🚀
Your progress is saved in this browser, on this device only.