Master the building blocks of deep learning: perceptrons, multi-layer networks, activation functions, backpropagation, and optimization algorithms. Build neural networks from first principles.
Neural networks are the foundation of modern AI—from GPT to self-driving cars. But what exactly is a "neural network"? How do they learn? And why do they work so well?
In this lesson, we'll build neural networks from first principles. You'll understand:
1. The Perceptron: The simplest neural network—a single neuron that learns linear boundaries
2. Multi-Layer Perceptrons: Stacking layers to solve complex, non-linear problems
3. Activation Functions: The "magic sauce" that gives neural networks their power
4. Backpropagation: The algorithm that enables learning (gradient computation via chain rule)
5. Optimizers: Algorithms that make training faster and more effective
You've already seen the mathematics (Lesson 2). Now you'll see how those concepts combine to create intelligent systems. By the end, you'll understand why neural networks are called "universal function approximators" and be able to build and train your own networks from scratch.
Ready to unlock the secrets of deep learning? Let's dive in! 🧠
The perceptron (1958) is the simplest neural network—a single artificial neuron. Understanding it is crucial for understanding modern deep learning.
A perceptron takes multiple inputs, applies weights, and produces an output:
1import numpy as np
2
3class Perceptron:
4 """Single neuron - the building block of neural networks"""
5
6 def __init__(self, input_size, learning_rate=0.01):
7 # Initialize weights randomly
8 self.weights = np.random.randn(input_size) * 0.1
9 self.bias = 0
10 self.lr = learning_rate
11
12 def predict(self, x):
13 """Forward pass: compute weighted sum + activation"""
14 # Weighted sum: z = w·x + b
15 z = np.dot(x, self.weights) + self.bias
16
17 # Step activation: output 1 if z > 0, else 0
18 return 1 if z >= 0 else 0
19
20 def train(self, X, y, epochs=100):
21 """Train using perceptron learning rule"""
22 for epoch in range(epochs):
23 errors = 0
24 for xi, yi in zip(X, y):
25 # Predict
26 prediction = self.predict(xi)
27
28 # Update if wrong (perceptron learning rule)
29 error = yi - prediction
30 if error != 0:
31 self.weights += self.lr * error * xi
32 self.bias += self.lr * error
33 errors += 1
34
35 if errors == 0:
36 print(f"Converged at epoch {epoch + 1}")
37 break
38
39# Example: AND gate
40print("Training Perceptron on AND Gate")
41print("-" * 40)
42
43X_and = np.array([
44 [0, 0],
45 [0, 1],
46 [1, 0],
47 [1, 1]
48])
49
50y_and = np.array([0, 0, 0, 1])
51
52perceptron = Perceptron(input_size=2, learning_rate=0.1)
53perceptron.train(X_and, y_and, epochs=20)
54
55print("\nTesting AND Gate:")
56for x, expected in zip(X_and, y_and):
57 pred = perceptron.predict(x)
58 print(f" {x} → {pred} (expected: {expected})")Output:
1Training Perceptron on AND Gate
2----------------------------------------
3Converged at epoch 6
4
5Testing AND Gate:
6 [0 0] → 0 (expected: 0)
7 [0 1] → 0 (expected: 0)
8 [1 0] → 0 (expected: 0)
9 [1 1] → 1 (expected: 1)1. Weighted Sum
1z = w₁·x₁ + w₂·x₂ + ... + wₙ·xₙ + b2. Activation Function (Step function)
1output = 1 if z ≥ 0 else 03. Learning Rule
1weight_update = learning_rate × error × input
2error = true_value - predicted_valueThe perceptron can only learn linearly separable problems. XOR is not linearly separable:
1# XOR gate - impossible for single perceptron!
2X_xor = np.array([
3 [0, 0], # → 0
4 [0, 1], # → 1
5 [1, 0], # → 1
6 [1, 1] # → 0
7])
8
9y_xor = np.array([0, 1, 1, 0])
10
11print("\nAttempting to learn XOR with single perceptron...")
12perceptron_xor = Perceptron(input_size=2, learning_rate=0.1)
13perceptron_xor.train(X_xor, y_xor, epochs=1000)
14
15print("\nTesting XOR:")
16correct = 0
17for x, expected in zip(X_xor, y_xor):
18 pred = perceptron_xor.predict(x)
19 correct += (pred == expected)
20 print(f" {x} → {pred} (expected: {expected})")
21
22print(f"\nAccuracy: {correct/len(y_xor)*100:.1f}%")
23print("❌ Single perceptron cannot solve XOR!")Why XOR Fails: You cannot draw a single straight line to separate the XOR outputs.
The Solution: Add hidden layers → Multi-Layer Perceptron (MLP)!
What Perceptrons Can Do:
What Perceptrons Cannot Do:
Historical Impact: The XOR problem led to the "AI winter" (1970s-1980s) until backpropagation made multi-layer networks practical.
By stacking perceptrons in layers, we create networks that can learn ANY function—this is the Universal Approximation Theorem.
1import numpy as np
2
3class MultiLayerPerceptron:
4 """Complete neural network with hidden layers"""
5
6 def __init__(self, layer_sizes, learning_rate=0.01):
7 """
8 layer_sizes: list of neurons per layer
9 Example: [2, 4, 3, 1] = 2 inputs, 4 hidden, 3 hidden, 1 output
10 """
11 self.layers = []
12 self.lr = learning_rate
13
14 # Initialize weights for each layer
15 for i in range(len(layer_sizes) - 1):
16 layer = {
17 'weights': np.random.randn(layer_sizes[i], layer_sizes[i+1]) * 0.1,
18 'bias': np.zeros(layer_sizes[i+1])
19 }
20 self.layers.append(layer)
21
22 print(f"Created MLP: {layer_sizes}")
23 print(f"Total parameters: {self.count_parameters()}")
24
25 def count_parameters(self):
26 """Count total trainable parameters"""
27 total = 0
28 for layer in self.layers:
29 total += layer['weights'].size + layer['bias'].size
30 return total
31
32 def sigmoid(self, z):
33 """Sigmoid activation: σ(z) = 1 / (1 + e^(-z))"""
34 return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
35
36 def sigmoid_derivative(self, a):
37 """Derivative: σ'(z) = σ(z) × (1 - σ(z))"""
38 return a * (1 - a)
39
40 def forward(self, X):
41 """Forward propagation through all layers"""
42 activations = [X] # Store activations for backprop
43
44 for i, layer in enumerate(self.layers):
45 # Weighted sum
46 z = np.dot(activations[-1], layer['weights']) + layer['bias']
47
48 # Activation
49 a = self.sigmoid(z)
50 activations.append(a)
51
52 return activations
53
54 def backward(self, activations, y):
55 """Backpropagation: compute gradients"""
56 m = y.shape[0]
57 gradients = []
58
59 # Output layer error
60 delta = activations[-1] - y
61
62 # Backpropagate through layers
63 for i in reversed(range(len(self.layers))):
64 # Gradients for this layer
65 dW = np.dot(activations[i].T, delta) / m
66 db = np.sum(delta, axis=0) / m
67
68 gradients.insert(0, {'dW': dW, 'db': db})
69
70 # Propagate error to previous layer
71 if i > 0:
72 delta = np.dot(delta, self.layers[i]['weights'].T) * \
73 self.sigmoid_derivative(activations[i])
74
75 return gradients
76
77 def update_weights(self, gradients):
78 """Update weights using computed gradients"""
79 for layer, grad in zip(self.layers, gradients):
80 layer['weights'] -= self.lr * grad['dW']
81 layer['bias'] -= self.lr * grad['db']
82
83 def train(self, X, y, epochs=1000, verbose=True):
84 """Train the network"""
85 for epoch in range(epochs):
86 # Forward pass
87 activations = self.forward(X)
88
89 # Compute loss
90 predictions = activations[-1]
91 loss = -np.mean(y * np.log(predictions + 1e-8) +
92 (1 - y) * np.log(1 - predictions + 1e-8))
93
94 # Backward pass
95 gradients = self.backward(activations, y)
96
97 # Update weights
98 self.update_weights(gradients)
99
100 # Print progress
101 if verbose and (epoch % 100 == 0 or epoch == epochs - 1):
102 accuracy = np.mean((predictions > 0.5) == y) * 100
103 print(f"Epoch {epoch:4d} | Loss: {loss:.6f} | Accuracy: {accuracy:.2f}%")
104
105 def predict(self, X):
106 """Make predictions"""
107 activations = self.forward(X)
108 return activations[-1]
109
110# Solve XOR with MLP!
111print("=" * 60)
112print("SOLVING XOR WITH MULTI-LAYER PERCEPTRON")
113print("=" * 60)
114print()
115
116X_xor = np.array([
117 [0, 0],
118 [0, 1],
119 [1, 0],
120 [1, 1]
121])
122
123y_xor = np.array([[0], [1], [1], [0]])
124
125# Create MLP: 2 inputs → 4 hidden → 1 output
126mlp = MultiLayerPerceptron(layer_sizes=[2, 4, 1], learning_rate=0.5)
127print()
128
129print("Training on XOR problem...")
130print("-" * 60)
131mlp.train(X_xor, y_xor, epochs=5000)
132print()
133
134print("Final Predictions:")
135print("-" * 40)
136predictions = mlp.predict(X_xor)
137for i, (x, pred, true) in enumerate(zip(X_xor, predictions, y_xor)):
138 correct = "✓" if (pred > 0.5) == true else "✗"
139 print(f"{x} → {pred[0]:.6f} (expected: {true[0]}) {correct}")Theorem: A neural network with:
Can approximate any continuous function to arbitrary precision.
Intuition:
Number of Layers:
Neurons per Layer:
Example Architectures:
1# Binary classification (10 features)
2model = MultiLayerPerceptron([10, 20, 10, 1])
3
4# Multi-class classification (100 features, 5 classes)
5model = MultiLayerPerceptron([100, 64, 32, 5])
6
7# Image classification (784 pixels = 28×28, 10 digits)
8model = MultiLayerPerceptron([784, 128, 64, 10])Power of Hidden Layers:
Depth vs Width:
Universal Approximation ≠ Easy Training: Theory says it's possible, but finding the right weights requires good optimization!
Activation functions are the "magic ingredient" that give neural networks their power. Without them, deep networks would collapse into simple linear models.
Problem: Stacking linear layers is still linear!
1# Without activation functions:
2h1 = W1 @ x + b1
3h2 = W2 @ h1 + b2
4output = W3 @ h2 + b3
5
6# Simplifies to:
7output = (W3 @ W2 @ W1) @ x + (combined biases)
8# This is just a single linear transformation!Solution: Add non-linear activation functions between layers.
1. Sigmoid (σ)
Squashes inputs to range [0, 1] — good for probabilities.
1def sigmoid(z):
2 return 1 / (1 + np.exp(-z))
3
4def sigmoid_derivative(a):
5 return a * (1 - a)
6
7# Test
8z = np.linspace(-6, 6, 100)
9a = sigmoid(z)
10
11print("Sigmoid Properties:")
12print(f" Range: [{a.min():.3f}, {a.max():.3f}]")
13print(f" At z=0: {sigmoid(0):.3f}")
14print(f" At z=5: {sigmoid(5):.6f} (saturated)")
15print(f" At z=-5: {sigmoid(-5):.6f} (saturated)")Pros:
Cons:
Use Case: Binary classification output layer
---
2. Tanh (Hyperbolic Tangent)
Squashes inputs to range [-1, 1] — zero-centered version of sigmoid.
1def tanh(z):
2 return np.tanh(z)
3
4def tanh_derivative(a):
5 return 1 - a**2
6
7# Test
8z = np.linspace(-6, 6, 100)
9a = tanh(z)
10
11print("Tanh Properties:")
12print(f" Range: [{a.min():.3f}, {a.max():.3f}]")
13print(f" At z=0: {tanh(0):.3f}")
14print(f" Zero-centered: Yes")Pros:
Cons:
Use Case: Hidden layers in RNNs
---
3. ReLU (Rectified Linear Unit) ⭐ MOST POPULAR
Simple yet powerful: max(0, z)
1def relu(z):
2 return np.maximum(0, z)
3
4def relu_derivative(z):
5 return (z > 0).astype(float)
6
7# Test
8z = np.array([-2, -1, 0, 1, 2, 5, 10])
9a = relu(z)
10
11print("ReLU Properties:")
12print(f" Input: {z}")
13print(f" Output: {a}")
14print(f" Gradient: {relu_derivative(z)}")Output:
1Input: [-2 -1 0 1 2 5 10]
2Output: [ 0 0 0 1 2 5 10]
3Gradient: [0. 0. 0. 1. 1. 1. 1.]Pros:
Cons:
Use Case: Default choice for hidden layers!
---
4. Leaky ReLU
Fixes dying ReLU problem: max(αz, z) where α ≈ 0.01
1def leaky_relu(z, alpha=0.01):
2 return np.where(z > 0, z, alpha * z)
3
4def leaky_relu_derivative(z, alpha=0.01):
5 return np.where(z > 0, 1, alpha)
6
7# Test
8z = np.array([-2, -1, 0, 1, 2])
9a = leaky_relu(z)
10
11print("Leaky ReLU (α=0.01):")
12print(f" Input: {z}")
13print(f" Output: {a}")
14print(f" Gradient: {leaky_relu_derivative(z)}")Output:
1Input: [-2 -1 0 1 2]
2Output: [-0.02 -0.01 0. 1. 2. ]
3Gradient: [0.01 0.01 0.01 1. 1. ]Pros:
Cons:
Use Case: When ReLU neurons are dying
---
5. Softmax (For Multi-Class Classification)
Converts logits to probability distribution.
1def softmax(z):
2 """Softmax: exponential normalization"""
3 exp_z = np.exp(z - np.max(z)) # Numerical stability
4 return exp_z / exp_z.sum(axis=-1, keepdims=True)
5
6# Test
7logits = np.array([2.0, 1.0, 0.1])
8probs = softmax(logits)
9
10print("Softmax Activation:")
11print(f" Logits: {logits}")
12print(f" Probabilities: {probs}")
13print(f" Sum: {probs.sum():.6f}")
14print(f" Max class: {np.argmax(probs)}")Output:
1Logits: [2. 1. 0.1]
2Probabilities: [0.659 0.242 0.099]
3Sum: 1.000000
4Max class: 0Use Case: Multi-class classification output layer
1Output Layer:
2├─ Binary classification → Sigmoid
3├─ Multi-class classification → Softmax
4└─ Regression → Linear (no activation)
5
6Hidden Layers:
7├─ Default choice → ReLU
8├─ ReLU dying? → Leaky ReLU / ELU
9├─ RNN/LSTM → Tanh
10└─ Need smooth gradients → Tanh1import matplotlib.pyplot as plt
2
3z = np.linspace(-5, 5, 200)
4
5fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
6
7# Plot activations
8ax1.plot(z, sigmoid(z), label='Sigmoid', linewidth=2)
9ax1.plot(z, tanh(z), label='Tanh', linewidth=2)
10ax1.plot(z, relu(z), label='ReLU', linewidth=2)
11ax1.plot(z, leaky_relu(z), label='Leaky ReLU', linewidth=2)
12ax1.axhline(0, color='black', linestyle='--', alpha=0.3)
13ax1.axvline(0, color='black', linestyle='--', alpha=0.3)
14ax1.set_xlabel('Input (z)', fontweight='bold')
15ax1.set_ylabel('Output', fontweight='bold')
16ax1.set_title('Activation Functions', fontweight='bold', fontsize=13)
17ax1.legend()
18ax1.grid(True, alpha=0.3)
19
20# Plot derivatives
21ax2.plot(z, sigmoid_derivative(sigmoid(z)), label='Sigmoid', linewidth=2)
22ax2.plot(z, tanh_derivative(tanh(z)), label='Tanh', linewidth=2)
23ax2.plot(z, relu_derivative(z), label='ReLU', linewidth=2)
24ax2.plot(z, leaky_relu_derivative(z), label='Leaky ReLU', linewidth=2)
25ax2.axhline(0, color='black', linestyle='--', alpha=0.3)
26ax2.set_xlabel('Input (z)', fontweight='bold')
27ax2.set_ylabel('Gradient', fontweight='bold')
28ax2.set_title('Activation Function Derivatives', fontweight='bold', fontsize=13)
29ax2.legend()
30ax2.grid(True, alpha=0.3)
31
32plt.tight_layout()
33plt.show()Why ReLU Dominates:
Vanishing Gradient Problem:
Modern Best Practice:
Backpropagation is the algorithm that makes neural networks learn. It efficiently computes gradients for all weights using the chain rule of calculus.
Training Loop:
The chain rule lets us decompose complex derivatives:
1If y = f(g(h(x))), then:
2dy/dx = (df/dg) × (dg/dh) × (dh/dx)Neural Network Example:
1Network: x → z₁=W₁x → a₁=σ(z₁) → z₂=W₂a₁ → a₂=σ(z₂) → Loss
2
3To find ∂Loss/∂W₁, we need:
4∂Loss/∂W₁ = (∂Loss/∂a₂) × (∂a₂/∂z₂) × (∂z₂/∂a₁) × (∂a₁/∂z₁) × (∂z₁/∂W₁)Let's implement backpropagation with detailed comments:
1import numpy as np
2
3class DetailedBackpropNetwork:
4 """Neural network with detailed backprop explanation"""
5
6 def __init__(self):
7 # Simple network: 2 → 3 → 1
8 np.random.seed(42)
9 self.W1 = np.random.randn(2, 3) * 0.5
10 self.b1 = np.zeros(3)
11 self.W2 = np.random.randn(3, 1) * 0.5
12 self.b2 = np.zeros(1)
13
14 print("Network: 2 inputs → 3 hidden → 1 output")
15 print(f"W1 shape: {self.W1.shape}")
16 print(f"W2 shape: {self.W2.shape}")
17
18 def sigmoid(self, z):
19 return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
20
21 def sigmoid_derivative(self, a):
22 return a * (1 - a)
23
24 def forward(self, X):
25 """Forward pass with intermediate storage"""
26 # Layer 1: Input → Hidden
27 self.z1 = np.dot(X, self.W1) + self.b1
28 self.a1 = self.sigmoid(self.z1)
29
30 # Layer 2: Hidden → Output
31 self.z2 = np.dot(self.a1, self.W2) + self.b2
32 self.a2 = self.sigmoid(self.z2)
33
34 return self.a2
35
36 def backward(self, X, y, learning_rate=0.1):
37 """Backpropagation with detailed steps"""
38 m = X.shape[0]
39
40 print("\n" + "="*60)
41 print("BACKPROPAGATION STEP-BY-STEP")
42 print("="*60)
43
44 # ==================== OUTPUT LAYER ====================
45 print("\n1. Output Layer Gradient")
46 print("-" * 40)
47
48 # Loss: Binary cross-entropy
49 # L = -[y·log(a₂) + (1-y)·log(1-a₂)]
50 loss = -np.mean(y * np.log(self.a2 + 1e-8) +
51 (1 - y) * np.log(1 - self.a2 + 1e-8))
52 print(f" Loss: {loss:.6f}")
53
54 # Gradient of loss w.r.t. output activation
55 # ∂L/∂a₂
56 dL_da2 = -(y / (self.a2 + 1e-8) - (1 - y) / (1 - self.a2 + 1e-8))
57 print(f" ∂L/∂a₂ shape: {dL_da2.shape}")
58
59 # Gradient of activation w.r.t. pre-activation
60 # ∂a₂/∂z₂ = σ'(z₂) = a₂(1-a₂)
61 da2_dz2 = self.sigmoid_derivative(self.a2)
62 print(f" ∂a₂/∂z₂ shape: {da2_dz2.shape}")
63
64 # Combined: ∂L/∂z₂ = ∂L/∂a₂ × ∂a₂/∂z₂ (chain rule!)
65 dL_dz2 = dL_da2 * da2_dz2
66 print(f" ∂L/∂z₂ = ∂L/∂a₂ × ∂a₂/∂z₂")
67 print(f" ∂L/∂z₂ shape: {dL_dz2.shape}")
68
69 # Gradient w.r.t. W2: ∂L/∂W₂ = a₁ᵀ × ∂L/∂z₂
70 dL_dW2 = np.dot(self.a1.T, dL_dz2) / m
71 dL_db2 = np.sum(dL_dz2, axis=0) / m
72 print(f" ∂L/∂W₂ shape: {dL_dW2.shape}")
73 print(f" ∂L/∂b₂ shape: {dL_db2.shape}")
74
75 # ==================== HIDDEN LAYER ====================
76 print("\n2. Hidden Layer Gradient (Backpropagate)")
77 print("-" * 40)
78
79 # Backpropagate error to hidden layer
80 # ∂L/∂a₁ = ∂L/∂z₂ × W₂ᵀ
81 dL_da1 = np.dot(dL_dz2, self.W2.T)
82 print(f" ∂L/∂a₁ = ∂L/∂z₂ @ W₂ᵀ")
83 print(f" ∂L/∂a₁ shape: {dL_da1.shape}")
84
85 # Apply chain rule through activation
86 # ∂L/∂z₁ = ∂L/∂a₁ × ∂a₁/∂z₁
87 da1_dz1 = self.sigmoid_derivative(self.a1)
88 dL_dz1 = dL_da1 * da1_dz1
89 print(f" ∂L/∂z₁ = ∂L/∂a₁ × σ'(z₁)")
90 print(f" ∂L/∂z₁ shape: {dL_dz1.shape}")
91
92 # Gradient w.r.t. W1: ∂L/∂W₁ = Xᵀ × ∂L/∂z₁
93 dL_dW1 = np.dot(X.T, dL_dz1) / m
94 dL_db1 = np.sum(dL_dz1, axis=0) / m
95 print(f" ∂L/∂W₁ shape: {dL_dW1.shape}")
96 print(f" ∂L/∂b₁ shape: {dL_db1.shape}")
97
98 # ==================== WEIGHT UPDATE ====================
99 print("\n3. Weight Update (Gradient Descent)")
100 print("-" * 40)
101
102 # Update weights: W = W - α × ∂L/∂W
103 self.W2 -= learning_rate * dL_dW2
104 self.b2 -= learning_rate * dL_db2
105 self.W1 -= learning_rate * dL_dW1
106 self.b1 -= learning_rate * dL_db1
107
108 print(f" W₂ -= {learning_rate} × ∂L/∂W₂")
109 print(f" W₁ -= {learning_rate} × ∂L/∂W₁")
110 print(f" Max W₂ change: {np.max(np.abs(learning_rate * dL_dW2)):.6f}")
111 print(f" Max W₁ change: {np.max(np.abs(learning_rate * dL_dW1)):.6f}")
112
113 return loss
114
115# Demonstrate backpropagation
116print("=" * 60)
117print("BACKPROPAGATION DEMONSTRATION")
118print("=" * 60)
119
120# XOR dataset
121X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
122y = np.array([[0], [1], [1], [0]])
123
124network = DetailedBackpropNetwork()
125
126print("\nInitial prediction (before training):")
127pred_before = network.forward(X)
128for i, (xi, yi, pi) in enumerate(zip(X, y, pred_before)):
129 print(f" {xi} → {pi[0]:.4f} (expected: {yi[0]})")
130
131# Single training step with detailed backprop
132print("\n" + "="*60)
133print("PERFORMING ONE TRAINING ITERATION")
134print("="*60)
135
136network.forward(X) # Forward pass
137loss = network.backward(X, y, learning_rate=1.0) # Detailed backward pass
138
139print("\n" + "="*60)
140print("AFTER ONE TRAINING STEP")
141print("="*60)
142
143pred_after = network.forward(X)
144for i, (xi, yi, pi) in enumerate(zip(X, y, pred_after)):
145 improvement = abs(pi[0] - yi[0]) < abs(pred_before[i][0] - yi[0])
146 symbol = "✓" if improvement else "→"
147 print(f" {xi} → {pi[0]:.4f} (expected: {yi[0]}) {symbol}")1. Chain Rule Everywhere
Each gradient is computed by multiplying upstream gradient with local gradient:
1∂Loss/∂W₁ = (∂Loss/∂a₂) × (∂a₂/∂z₂) × (∂z₂/∂a₁) × (∂a₁/∂z₁) × (∂z₁/∂W₁)2. Backward Flow
3. Matrix Transposes
1# Forward: a₁ = X @ W₁
2# Backward: ∂L/∂X = ∂L/∂a₁ @ W₁ᵀ
3
4# Forward: z₂ = a₁ @ W₂
5# Backward: ∂L/∂a₁ = ∂L/∂z₂ @ W₂ᵀ4. Efficiency
5. Activation Functions Matter
1Forward Pass:
2X → [×W₁] → z₁ → [σ] → a₁ → [×W₂] → z₂ → [σ] → a₂ → [Loss]
3
4Backward Pass:
5X ← [W₁ᵀ×] ← ∂z₁ ← [σ'×] ← ∂a₁ ← [W₂ᵀ×] ← ∂z₂ ← [σ'×] ← ∂a₂ ← [∂Loss]Why This Matters:
Gradient descent has many variants. Modern optimizers like Adam make training faster, more stable, and achieve better results.
Training neural networks = navigating a high-dimensional loss surface to find the minimum.
The baseline optimizer—pure gradient descent.
1class SGD:
2 """Basic stochastic gradient descent"""
3
4 def __init__(self, learning_rate=0.01):
5 self.lr = learning_rate
6
7 def update(self, param, grad):
8 """Update rule: θ = θ - α·∇θ"""
9 return param - self.lr * grad
10
11# Usage
12optimizer = SGD(learning_rate=0.01)
13W_new = optimizer.update(W, dL_dW)Pros:
Cons:
---
Accumulates past gradients to smooth updates and accelerate in consistent directions.
1class SGDMomentum:
2 """SGD with momentum"""
3
4 def __init__(self, learning_rate=0.01, momentum=0.9):
5 self.lr = learning_rate
6 self.momentum = momentum
7 self.velocity = {}
8
9 def update(self, param_name, param, grad):
10 """
11 Update rule:
12 v = β·v - α·∇θ
13 θ = θ + v
14 """
15 # Initialize velocity on first call
16 if param_name not in self.velocity:
17 self.velocity[param_name] = np.zeros_like(param)
18
19 # Accumulate velocity
20 self.velocity[param_name] = (self.momentum * self.velocity[param_name] -
21 self.lr * grad)
22
23 # Update parameter
24 return param + self.velocity[param_name]
25
26# Example: Ball rolling down hill (momentum helps!)
27def f(x):
28 return 0.5 * x**2 + 0.1 * np.sin(10 * x) # Loss with local minima
29
30def f_grad(x):
31 return x + np.cos(10 * x)
32
33# SGD vs SGD+Momentum
34x_sgd = 3.0
35x_momentum = 3.0
36
37sgd = SGD(learning_rate=0.01)
38sgd_mom = SGDMomentum(learning_rate=0.01, momentum=0.9)
39
40history_sgd = [x_sgd]
41history_momentum = [x_momentum]
42
43for _ in range(100):
44 # SGD
45 x_sgd = sgd.update(x_sgd, f_grad(x_sgd))
46 history_sgd.append(x_sgd)
47
48 # SGD + Momentum
49 x_momentum = sgd_mom.update('x', x_momentum, f_grad(x_momentum))
50 history_momentum.append(x_momentum)
51
52print(f"Final position:")
53print(f" SGD: x = {x_sgd:.4f}, f(x) = {f(x_sgd):.4f}")
54print(f" SGD+Momentum: x = {x_momentum:.4f}, f(x) = {f(x_momentum):.4f}")
55print(f"\nMomentum helps escape local minima and converge faster!")Pros:
Cons:
---
Adapts learning rate per parameter based on recent gradient magnitudes.
1class RMSprop:
2 """RMSprop optimizer"""
3
4 def __init__(self, learning_rate=0.001, decay=0.9, epsilon=1e-8):
5 self.lr = learning_rate
6 self.decay = decay
7 self.epsilon = epsilon
8 self.cache = {}
9
10 def update(self, param_name, param, grad):
11 """
12 Update rule:
13 cache = decay·cache + (1-decay)·∇²
14 θ = θ - α·∇ / (√cache + ε)
15 """
16 # Initialize cache on first call
17 if param_name not in self.cache:
18 self.cache[param_name] = np.zeros_like(param)
19
20 # Accumulate squared gradients
21 self.cache[param_name] = (self.decay * self.cache[param_name] +
22 (1 - self.decay) * grad**2)
23
24 # Adaptive learning rate
25 update = self.lr * grad / (np.sqrt(self.cache[param_name]) + self.epsilon)
26
27 return param - update
28
29# Example: Different parameters need different learning rates
30# Parameter 1: steep gradient (needs small LR)
31# Parameter 2: gentle gradient (needs large LR)
32
33W1, W2 = 5.0, 5.0 # Start at same position
34
35rmsprop = RMSprop(learning_rate=0.1)
36
37for i in range(10):
38 # Steep gradient for W1
39 grad1 = 10.0 * W1
40 W1 = rmsprop.update('W1', W1, grad1)
41
42 # Gentle gradient for W2
43 grad2 = 0.1 * W2
44 W2 = rmsprop.update('W2', W2, grad2)
45
46 if i % 3 == 0:
47 print(f"Step {i}: W1={W1:.4f}, W2={W2:.4f}")
48
49print("\nRMSprop automatically adjusts learning rates!")
50print("Steep gradients → smaller effective LR")
51print("Gentle gradients → larger effective LR")Pros:
Cons:
---
Combines momentum and RMSprop—the default optimizer for most problems.
1class Adam:
2 """Adam optimizer - combines momentum and RMSprop"""
3
4 def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
5 self.lr = learning_rate
6 self.beta1 = beta1 # Momentum decay
7 self.beta2 = beta2 # RMSprop decay
8 self.epsilon = epsilon
9 self.m = {} # First moment (momentum)
10 self.v = {} # Second moment (RMSprop)
11 self.t = 0 # Time step
12
13 def update(self, param_name, param, grad):
14 """
15 Update rule:
16 m = β₁·m + (1-β₁)·∇ (momentum)
17 v = β₂·v + (1-β₂)·∇² (RMSprop)
18 m̂ = m / (1-β₁ᵗ) (bias correction)
19 v̂ = v / (1-β₂ᵗ) (bias correction)
20 θ = θ - α·m̂ / (√v̂ + ε)
21 """
22 # Initialize on first call
23 if param_name not in self.m:
24 self.m[param_name] = np.zeros_like(param)
25 self.v[param_name] = np.zeros_like(param)
26
27 self.t += 1
28
29 # Update biased first moment (momentum)
30 self.m[param_name] = (self.beta1 * self.m[param_name] +
31 (1 - self.beta1) * grad)
32
33 # Update biased second moment (RMSprop)
34 self.v[param_name] = (self.beta2 * self.v[param_name] +
35 (1 - self.beta2) * grad**2)
36
37 # Bias correction (important for early steps!)
38 m_hat = self.m[param_name] / (1 - self.beta1**self.t)
39 v_hat = self.v[param_name] / (1 - self.beta2**self.t)
40
41 # Update parameter
42 update = self.lr * m_hat / (np.sqrt(v_hat) + self.epsilon)
43
44 return param - update
45
46# Compare all optimizers
47print("=" * 60)
48print("OPTIMIZER COMPARISON")
49print("=" * 60)
50
51def rosenbrock(x, y):
52 """Rosenbrock function - classic optimization test"""
53 return (1 - x)**2 + 100 * (y - x**2)**2
54
55def rosenbrock_grad(x, y):
56 dx = -2 * (1 - x) - 400 * x * (y - x**2)
57 dy = 200 * (y - x**2)
58 return np.array([dx, dy])
59
60# Initialize optimizers
61optimizers = {
62 'SGD': SGD(learning_rate=0.001),
63 'SGD+Momentum': SGDMomentum(learning_rate=0.001, momentum=0.9),
64 'RMSprop': RMSprop(learning_rate=0.01),
65 'Adam': Adam(learning_rate=0.01)
66}
67
68# Starting point
69start = np.array([-1.0, 1.0])
70positions = {name: [start.copy()] for name in optimizers}
71
72# Optimize
73for step in range(500):
74 for name, opt in optimizers.items():
75 pos = positions[name][-1]
76 grad = rosenbrock_grad(pos[0], pos[1])
77
78 if name == 'SGD':
79 new_pos = opt.update(pos, grad)
80 else:
81 new_pos = np.array([
82 opt.update(f'{name}_x', pos[0], grad[0]),
83 opt.update(f'{name}_y', pos[1], grad[1])
84 ])
85
86 positions[name].append(new_pos)
87
88# Results
89print("\nFinal Results (starting at [-1, 1], goal: [1, 1]):")
90print("-" * 60)
91for name in optimizers:
92 final = positions[name][-1]
93 loss = rosenbrock(final[0], final[1])
94 print(f"{name:15} → [{final[0]:.4f}, {final[1]:.4f}] Loss: {loss:.6f}")
95
96print("\nAdam typically converges fastest and most reliably!")Pros:
Cons:
---
| Optimizer | Speed | Memory | When to Use |
|-----------|-------|--------|-------------|
| SGD | Slow | Low | Simple problems, research |
| SGD+Momentum | Medium | Low | When SGD oscillates |
| RMSprop | Fast | Medium | RNNs, variable gradients |
| Adam | Fast | Medium | Default choice |
Even with Adam, learning rate can be reduced over time:
1# Step decay: Reduce LR every N epochs
2def step_decay(initial_lr, epoch, drop_every=10, factor=0.5):
3 return initial_lr * (factor ** (epoch // drop_every))
4
5# Exponential decay
6def exp_decay(initial_lr, epoch, decay_rate=0.95):
7 return initial_lr * (decay_rate ** epoch)
8
9# Cosine annealing
10def cosine_decay(initial_lr, epoch, total_epochs):
11 return initial_lr * 0.5 * (1 + np.cos(np.pi * epoch / total_epochs))Why Optimizers Matter:
Modern Best Practice:
Rule of Thumb:
Implement a complete multi-layer neural network with multiple activation functions and optimizers. Train on a challenging spiral dataset that requires deep non-linear boundaries.
1# Complete Neural Network with Multiple Optimizers
2# Comparing SGD, Momentum, RMSprop, and Adam on Spiral Dataset
3
4import numpy as np
5import matplotlib.pyplot as plt
6from matplotlib.colors import ListedColormap
7
8print("=" * 80)
9print("DEEP LEARNING: SPIRAL CLASSIFICATION WITH NEURAL NETWORKS")
10print("=" * 80)
11print()
12
13# ============================================================================
14# PART 1: Generate Spiral Dataset (Non-linearly Separable)
15# ============================================================================
16
17def generate_spiral_data(n_points=100, n_classes=3, noise=0.1):
18 """Generate spiral dataset - classic neural network challenge"""
19 X = np.zeros((n_points * n_classes, 2))
20 y = np.zeros(n_points * n_classes, dtype=int)
21
22 for class_idx in range(n_classes):
23 ix = range(n_points * class_idx, n_points * (class_idx + 1))
24 r = np.linspace(0.0, 1, n_points) # Radius
25 t = np.linspace(class_idx * 4, (class_idx + 1) * 4, n_points) + \
26 np.random.randn(n_points) * noise # Theta with noise
27
28 X[ix] = np.c_[r * np.sin(t * 2.5), r * np.cos(t * 2.5)]
29 y[ix] = class_idx
30
31 return X, y
32
33# Generate data
34X, y = generate_spiral_data(n_points=100, n_classes=3, noise=0.2)
35
36# One-hot encode labels
37n_classes = 3
38y_onehot = np.eye(n_classes)[y]
39
40print("Dataset Statistics:")
41print(f" Samples: {len(X)}")
42print(f" Features: {X.shape[1]}")
43print(f" Classes: {n_classes}")
44print(f" Class distribution: {np.bincount(y)}")
45print()
46
47# Visualize dataset
48plt.figure(figsize=(8, 8))
49plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap='viridis',
50 edgecolors='black', linewidth=0.5)
51plt.title('Spiral Dataset - Non-linearly Separable',
52 fontweight='bold', fontsize=14)
53plt.xlabel('Feature 1', fontweight='bold')
54plt.ylabel('Feature 2', fontweight='bold')
55plt.colorbar(label='Class')
56plt.grid(True, alpha=0.3)
57plt.show()
58
59# ============================================================================
60# PART 2: Neural Network Implementation
61# ============================================================================
62
63class NeuralNetwork:
64 """Deep neural network with configurable architecture and optimizer"""
65
66 def __init__(self, layer_sizes, activation='relu', optimizer='adam',
67 learning_rate=0.01):
68 self.layer_sizes = layer_sizes
69 self.activation_name = activation
70 self.optimizer_name = optimizer
71 self.lr = learning_rate
72
73 # Initialize weights
74 self.weights = []
75 self.biases = []
76
77 for i in range(len(layer_sizes) - 1):
78 # He initialization for ReLU, Xavier for others
79 if activation == 'relu':
80 w = np.random.randn(layer_sizes[i], layer_sizes[i+1]) * \
81 np.sqrt(2.0 / layer_sizes[i])
82 else:
83 w = np.random.randn(layer_sizes[i], layer_sizes[i+1]) * \
84 np.sqrt(1.0 / layer_sizes[i])
85
86 b = np.zeros(layer_sizes[i+1])
87 self.weights.append(w)
88 self.biases.append(b)
89
90 # Initialize optimizer state
91 self.init_optimizer()
92
93 print(f"Neural Network Created:")
94 print(f" Architecture: {layer_sizes}")
95 print(f" Activation: {activation}")
96 print(f" Optimizer: {optimizer}")
97 print(f" Learning rate: {learning_rate}")
98 print(f" Total parameters: {self.count_parameters()}")
99
100 def count_parameters(self):
101 total = sum(w.size + b.size for w, b in zip(self.weights, self.biases))
102 return total
103
104 def init_optimizer(self):
105 """Initialize optimizer-specific variables"""
106 if self.optimizer_name == 'momentum':
107 self.velocities_w = [np.zeros_like(w) for w in self.weights]
108 self.velocities_b = [np.zeros_like(b) for b in self.biases]
109 self.momentum = 0.9
110
111 elif self.optimizer_name == 'rmsprop':
112 self.cache_w = [np.zeros_like(w) for w in self.weights]
113 self.cache_b = [np.zeros_like(b) for b in self.biases]
114 self.decay = 0.9
115 self.epsilon = 1e-8
116
117 elif self.optimizer_name == 'adam':
118 self.m_w = [np.zeros_like(w) for w in self.weights]
119 self.m_b = [np.zeros_like(b) for b in self.biases]
120 self.v_w = [np.zeros_like(w) for w in self.weights]
121 self.v_b = [np.zeros_like(b) for b in self.biases]
122 self.beta1 = 0.9
123 self.beta2 = 0.999
124 self.epsilon = 1e-8
125 self.t = 0
126
127 # Activation functions
128 def relu(self, z):
129 return np.maximum(0, z)
130
131 def relu_derivative(self, z):
132 return (z > 0).astype(float)
133
134 def sigmoid(self, z):
135 return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
136
137 def sigmoid_derivative(self, a):
138 return a * (1 - a)
139
140 def softmax(self, z):
141 exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
142 return exp_z / np.sum(exp_z, axis=1, keepdims=True)
143
144 def forward(self, X):
145 """Forward pass through network"""
146 self.z_values = []
147 self.activations = [X]
148
149 for i in range(len(self.weights)):
150 # Linear transformation
151 z = np.dot(self.activations[-1], self.weights[i]) + self.biases[i]
152 self.z_values.append(z)
153
154 # Activation
155 if i == len(self.weights) - 1:
156 # Output layer: softmax
157 a = self.softmax(z)
158 else:
159 # Hidden layers: chosen activation
160 if self.activation_name == 'relu':
161 a = self.relu(z)
162 else:
163 a = self.sigmoid(z)
164
165 self.activations.append(a)
166
167 return self.activations[-1]
168
169 def backward(self, X, y):
170 """Backward pass: compute gradients"""
171 m = X.shape[0]
172 n_layers = len(self.weights)
173
174 gradients_w = []
175 gradients_b = []
176
177 # Output layer gradient (softmax + cross-entropy)
178 delta = self.activations[-1] - y
179
180 # Backpropagate through layers
181 for i in reversed(range(n_layers)):
182 # Gradients for this layer
183 dW = np.dot(self.activations[i].T, delta) / m
184 db = np.sum(delta, axis=0) / m
185
186 gradients_w.insert(0, dW)
187 gradients_b.insert(0, db)
188
189 # Propagate to previous layer
190 if i > 0:
191 delta = np.dot(delta, self.weights[i].T)
192
193 # Apply activation derivative
194 if self.activation_name == 'relu':
195 delta *= self.relu_derivative(self.z_values[i-1])
196 else:
197 delta *= self.sigmoid_derivative(self.activations[i])
198
199 return gradients_w, gradients_b
200
201 def update_weights(self, gradients_w, gradients_b):
202 """Update weights using specified optimizer"""
203 if self.optimizer_name == 'sgd':
204 for i in range(len(self.weights)):
205 self.weights[i] -= self.lr * gradients_w[i]
206 self.biases[i] -= self.lr * gradients_b[i]
207
208 elif self.optimizer_name == 'momentum':
209 for i in range(len(self.weights)):
210 self.velocities_w[i] = self.momentum * self.velocities_w[i] - \
211 self.lr * gradients_w[i]
212 self.velocities_b[i] = self.momentum * self.velocities_b[i] - \
213 self.lr * gradients_b[i]
214
215 self.weights[i] += self.velocities_w[i]
216 self.biases[i] += self.velocities_b[i]
217
218 elif self.optimizer_name == 'rmsprop':
219 for i in range(len(self.weights)):
220 self.cache_w[i] = self.decay * self.cache_w[i] + \
221 (1 - self.decay) * gradients_w[i]**2
222 self.cache_b[i] = self.decay * self.cache_b[i] + \
223 (1 - self.decay) * gradients_b[i]**2
224
225 self.weights[i] -= self.lr * gradients_w[i] / \
226 (np.sqrt(self.cache_w[i]) + self.epsilon)
227 self.biases[i] -= self.lr * gradients_b[i] / \
228 (np.sqrt(self.cache_b[i]) + self.epsilon)
229
230 elif self.optimizer_name == 'adam':
231 self.t += 1
232
233 for i in range(len(self.weights)):
234 # Update biased first moment
235 self.m_w[i] = self.beta1 * self.m_w[i] + \
236 (1 - self.beta1) * gradients_w[i]
237 self.m_b[i] = self.beta1 * self.m_b[i] + \
238 (1 - self.beta1) * gradients_b[i]
239
240 # Update biased second moment
241 self.v_w[i] = self.beta2 * self.v_w[i] + \
242 (1 - self.beta2) * gradients_w[i]**2
243 self.v_b[i] = self.beta2 * self.v_b[i] + \
244 (1 - self.beta2) * gradients_b[i]**2
245
246 # Bias correction
247 m_w_hat = self.m_w[i] / (1 - self.beta1**self.t)
248 m_b_hat = self.m_b[i] / (1 - self.beta1**self.t)
249 v_w_hat = self.v_w[i] / (1 - self.beta2**self.t)
250 v_b_hat = self.v_b[i] / (1 - self.beta2**self.t)
251
252 # Update parameters
253 self.weights[i] -= self.lr * m_w_hat / \
254 (np.sqrt(v_w_hat) + self.epsilon)
255 self.biases[i] -= self.lr * m_b_hat / \
256 (np.sqrt(v_b_hat) + self.epsilon)
257
258 def compute_loss(self, y_true, y_pred):
259 """Cross-entropy loss"""
260 m = y_true.shape[0]
261 loss = -np.sum(y_true * np.log(y_pred + 1e-8)) / m
262 return loss
263
264 def train(self, X, y, epochs=1000, verbose=True):
265 """Train the network"""
266 history = {'loss': [], 'accuracy': []}
267
268 for epoch in range(epochs):
269 # Forward pass
270 predictions = self.forward(X)
271
272 # Compute loss
273 loss = self.compute_loss(y, predictions)
274
275 # Compute accuracy
276 pred_classes = np.argmax(predictions, axis=1)
277 true_classes = np.argmax(y, axis=1)
278 accuracy = np.mean(pred_classes == true_classes)
279
280 history['loss'].append(loss)
281 history['accuracy'].append(accuracy)
282
283 # Backward pass
284 gradients_w, gradients_b = self.backward(X, y)
285
286 # Update weights
287 self.update_weights(gradients_w, gradients_b)
288
289 # Print progress
290 if verbose and (epoch % 100 == 0 or epoch == epochs - 1):
291 print(f"Epoch {epoch:4d} | Loss: {loss:.6f} | Accuracy: {accuracy*100:.2f}%")
292
293 return history
294
295# ============================================================================
296# PART 3: Compare Optimizers
297# ============================================================================
298
299print("\n" + "=" * 80)
300print("TRAINING WITH DIFFERENT OPTIMIZERS")
301print("=" * 80)
302print()
303
304optimizers = ['sgd', 'momentum', 'rmsprop', 'adam']
305networks = {}
306histories = {}
307
308# Architecture: 2 → 16 → 8 → 3
309layer_sizes = [2, 16, 8, 3]
310
311for opt in optimizers:
312 print(f"\nTraining with {opt.upper()}:")
313 print("-" * 60)
314
315 # Adjust learning rate per optimizer
316 if opt == 'sgd':
317 lr = 0.5
318 elif opt == 'momentum':
319 lr = 0.3
320 else:
321 lr = 0.01
322
323 network = NeuralNetwork(layer_sizes, activation='relu',
324 optimizer=opt, learning_rate=lr)
325
326 history = network.train(X, y_onehot, epochs=2000, verbose=False)
327
328 networks[opt] = network
329 histories[opt] = history
330
331 # Final accuracy
332 final_acc = history['accuracy'][-1]
333 final_loss = history['loss'][-1]
334 print(f"Final: Loss={final_loss:.6f}, Accuracy={final_acc*100:.2f}%")
335
336# ============================================================================
337# PART 4: Visualizations
338# ============================================================================
339
340print("\n" + "=" * 80)
341print("GENERATING VISUALIZATIONS")
342print("=" * 80)
343print()
344
345# Plot 1: Training curves
346fig = plt.figure(figsize=(18, 5))
347
348# Loss curves
349ax1 = fig.add_subplot(131)
350for opt in optimizers:
351 ax1.plot(histories[opt]['loss'], label=opt.upper(), linewidth=2)
352
353ax1.set_xlabel('Epoch', fontweight='bold', fontsize=11)
354ax1.set_ylabel('Loss', fontweight='bold', fontsize=11)
355ax1.set_title('Training Loss Comparison', fontweight='bold', fontsize=13)
356ax1.legend(fontsize=10)
357ax1.grid(True, alpha=0.3)
358
359# Accuracy curves
360ax2 = fig.add_subplot(132)
361for opt in optimizers:
362 ax2.plot([acc * 100 for acc in histories[opt]['accuracy']],
363 label=opt.upper(), linewidth=2)
364
365ax2.set_xlabel('Epoch', fontweight='bold', fontsize=11)
366ax2.set_ylabel('Accuracy (%)', fontweight='bold', fontsize=11)
367ax2.set_title('Training Accuracy Comparison', fontweight='bold', fontsize=13)
368ax2.legend(fontsize=10)
369ax2.grid(True, alpha=0.3)
370
371# Final comparison
372ax3 = fig.add_subplot(133)
373final_accs = [histories[opt]['accuracy'][-1] * 100 for opt in optimizers]
374colors = ['#FF6B6B', '#4ECDC4', '#45B7D1', '#96CEB4']
375bars = ax3.bar(optimizers, final_accs, color=colors, edgecolor='black', linewidth=2)
376
377for bar, acc in zip(bars, final_accs):
378 height = bar.get_height()
379 ax3.text(bar.get_x() + bar.get_width()/2., height,
380 f'{acc:.1f}%', ha='center', va='bottom', fontweight='bold')
381
382ax3.set_ylabel('Final Accuracy (%)', fontweight='bold', fontsize=11)
383ax3.set_title('Final Performance Comparison', fontweight='bold', fontsize=13)
384ax3.set_ylim([0, 105])
385ax3.grid(True, alpha=0.3, axis='y')
386
387plt.tight_layout()
388plt.show()
389
390# Plot 2: Decision boundaries
391fig, axes = plt.subplots(2, 2, figsize=(14, 14))
392axes = axes.ravel()
393
394h = 0.02 # Step size
395x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
396y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
397xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
398 np.arange(y_min, y_max, h))
399
400for idx, opt in enumerate(optimizers):
401 # Predict for mesh
402 Z = networks[opt].forward(np.c_[xx.ravel(), yy.ravel()])
403 Z = np.argmax(Z, axis=1)
404 Z = Z.reshape(xx.shape)
405
406 # Plot decision boundary
407 axes[idx].contourf(xx, yy, Z, alpha=0.4, cmap='viridis', levels=2)
408 axes[idx].scatter(X[:, 0], X[:, 1], c=y, s=40, cmap='viridis',
409 edgecolors='black', linewidth=1)
410
411 acc = histories[opt]['accuracy'][-1]
412 axes[idx].set_title(f'{opt.upper()} - Accuracy: {acc*100:.2f}%',
413 fontweight='bold', fontsize=12)
414 axes[idx].set_xlabel('Feature 1', fontweight='bold')
415 axes[idx].set_ylabel('Feature 2', fontweight='bold')
416 axes[idx].grid(True, alpha=0.3)
417
418plt.tight_layout()
419plt.show()
420
421# ============================================================================
422# SUMMARY
423# ============================================================================
424
425print("\n" + "=" * 80)
426print("KEY INSIGHTS FROM NEURAL NETWORK TRAINING")
427print("=" * 80)
428print()
429
430print("1️⃣ ARCHITECTURE")
431print(" ✓ Multi-layer network solves non-linear spiral problem")
432print(" ✓ Hidden layers learn hierarchical feature representations")
433print(" ✓ ReLU activation enables deep learning")
434print()
435
436print("2️⃣ OPTIMIZER COMPARISON")
437print(" ✓ SGD: Baseline, slowest convergence")
438print(" ✓ Momentum: Accelerates and smooths updates")
439print(" ✓ RMSprop: Adaptive learning rates per parameter")
440print(" ✓ Adam: Best overall performance (combines momentum + adaptive LR)")
441print()
442
443print("3️⃣ TRAINING DYNAMICS")
444print(" ✓ Adam converges fastest to high accuracy")
445print(" ✓ SGD can get stuck or oscillate")
446print(" ✓ Momentum helps escape local minima")
447print()
448
449print("4️⃣ DECISION BOUNDARIES")
450print(" ✓ Neural networks learn complex non-linear boundaries")
451print(" ✓ Better optimizers → cleaner decision regions")
452print(" ✓ Deep networks approximate any function (universal approximation)")
453print()
454
455print("=" * 80)
456print("🧠 You've mastered neural network fundamentals!")
457print(" Next: Deep learning frameworks (TensorFlow/PyTorch) automate this!")
458print("=" * 80)This comprehensive example demonstrates all neural network fundamentals by training on a challenging spiral classification problem.
1. Non-Linear Problem (Spiral Dataset)
2. Complete Neural Network Implementation
Architecture:
Forward Pass:
1z₁ = X @ W₁ + b₁
2a₁ = ReLU(z₁)
3z₂ = a₁ @ W₂ + b₂
4a₂ = ReLU(z₂)
5z₃ = a₂ @ W₃ + b₃
6output = Softmax(z₃)Backward Pass:
3. Four Optimizer Implementations
SGD: Basic gradient descent
1W = W - α·∇WMomentum: Accumulates velocity
1v = β·v - α·∇W
2W = W + vRMSprop: Adaptive learning rates
1cache = β·cache + (1-β)·∇²
2W = W - α·∇W / √(cache + ε)Adam: Best of both worlds
1m = β₁·m + (1-β₁)·∇W
2v = β₂·v + (1-β₂)·∇²
3W = W - α·m̂ / √(v̂ + ε)Training Speed:
Final Accuracy:
Decision Boundaries:
1. Universal Approximation in Action
The network learns to separate spirals—a highly non-linear problem. This demonstrates the power of:
2. Optimizer Comparison
Visual proof that optimizer choice matters:
3. Real-World Parallel
This spiral problem is analogous to:
Linear Algebra:
Calculus:
Optimization:
This example bridges theory and practice—showing how mathematical concepts combine to create intelligent systems!
Exceptional work! You now understand the fundamental building blocks of all deep learning systems. You've seen how perceptrons evolved into multi-layer networks, how activation functions enable non-linearity, how backpropagation computes gradients, and how modern optimizers accelerate training.
Next: In Lesson 6: Introduction to TensorFlow and Keras, we'll use these concepts with Google's powerful deep learning framework. You'll learn:
No more implementing backpropagation by hand—frameworks automate everything while you focus on architecture and results!
Your Deep Learning Journey Continues → Ready to build production-ready neural networks? Let's go! 🚀
Your progress is saved in this browser, on this device only.