Master the essential mathematical foundations of AI: linear algebra, calculus, and probability. Learn how vectors, matrices, and gradients power every neural network.
Ever wondered what actually happens inside a neural network? The answer is mathematics. But don't worry—you don't need to be a math genius to understand AI. In this lesson, we'll build intuition for the three mathematical pillars that power every AI system:
1. Linear Algebra: How AI represents and transforms data (vectors, matrices, tensors)
2. Calculus: How AI learns from mistakes (derivatives, gradients, optimization)
3. Probability: How AI makes decisions under uncertainty (distributions, Bayes' theorem)
You've already seen AI work in Lesson 1. Now you'll understand the mathematical magic behind it. We'll use visual examples, Python code, and intuitive explanations—no PhD required!
By the end of this lesson, you'll see that AI mathematics is not about memorizing formulas—it's about understanding elegant patterns that let machines learn.
Neural networks operate on numbers organized in structures. Understanding these structures is essential for AI.
Scalar: A single number
1temperature = 25.5 # Scalar: one valueVector: An array of numbers (1D)
1import numpy as np
2
3# RGB color as a vector
4color = np.array([255, 128, 64]) # Red, Green, Blue
5
6# Word embedding in AI (representing "king")
7word_vector = np.array([0.2, -0.5, 0.8, 1.2, -0.3])Vectors represent:
Matrix: A 2D grid of numbers
1# Grayscale image (5x5 pixels)
2image = np.array([
3 [0.1, 0.2, 0.3, 0.4, 0.5],
4 [0.2, 0.3, 0.4, 0.5, 0.6],
5 [0.3, 0.4, 0.5, 0.6, 0.7],
6 [0.4, 0.5, 0.6, 0.7, 0.8],
7 [0.5, 0.6, 0.7, 0.8, 0.9]
8])
9
10print(f"Image shape: {image.shape}") # (5, 5)Matrices represent:
Tensor: Multi-dimensional array (3D, 4D, etc.)
1# Color image: 32x32 pixels with RGB channels
2color_image = np.random.rand(32, 32, 3)
3print(f"Tensor shape: {color_image.shape}") # (32, 32, 3)
4
5# Batch of 10 color images
6batch = np.random.rand(10, 32, 32, 3)
7print(f"Batch shape: {batch.shape}") # (10, 32, 32, 3)Tensors represent:
1. Matrix Multiplication (The Core of Neural Networks)
Every layer in a neural network performs matrix multiplication:
1# Input: 3 features
2input_data = np.array([1.0, 2.0, 3.0]) # Shape: (3,)
3
4# Weights: 3 inputs → 2 outputs (neurons)
5weights = np.array([
6 [0.5, 0.8], # Neuron 1 weights
7 [0.3, -0.2], # Neuron 2 weights
8 [0.7, 0.4] # Neuron 3 weights
9]) # Shape: (3, 2)
10
11# Forward pass: Compute neuron outputs
12output = np.dot(input_data, weights)
13print(f"Neuron outputs: {output}")
14# Result: [3.6, 1.8]
15
16# What happened:
17# Neuron 1: 1.0*0.5 + 2.0*0.3 + 3.0*0.7 = 3.6
18# Neuron 2: 1.0*0.8 + 2.0*(-0.2) + 3.0*0.4 = 1.8This is how information flows through a neural network! Each neuron computes a weighted sum of its inputs.
2. Element-wise Operations (Activation Functions)
1# ReLU activation: max(0, x)
2def relu(x):
3 return np.maximum(0, x)
4
5values = np.array([-2, -1, 0, 1, 2])
6activated = relu(values)
7print(f"After ReLU: {activated}")
8# Output: [0, 0, 0, 1, 2]3. Transpose (Flipping Dimensions)
1matrix = np.array([[1, 2, 3], [4, 5, 6]])
2print(f"Original shape: {matrix.shape}") # (2, 3)
3
4transposed = matrix.T
5print(f"Transposed shape: {transposed.shape}") # (3, 2)
6print(transposed)
7# [[1, 4],
8# [2, 5],
9# [3, 6]]Used in backpropagation to propagate gradients backward through the network.
Every Neural Network Operation is Linear Algebra:
Key Insight: GPUs are optimized for matrix operations, which is why they're 100x faster than CPUs for deep learning!
Neural networks learn by minimizing errors. Calculus tells us how to adjust weights to reduce those errors.
A derivative measures how much a function's output changes when its input changes.
1# Simple function: f(x) = x^2
2def f(x):
3 return x ** 2
4
5# Derivative: f'(x) = 2x
6def f_derivative(x):
7 return 2 * x
8
9# At x = 3
10x = 3
11print(f"f({x}) = {f(x)}") # f(3) = 9
12print(f"f'({x}) = {f_derivative(x)}") # f'(3) = 6
13
14# Meaning: At x=3, if we increase x slightly,
15# f(x) increases 6 times fasterNeural networks have thousands of weights. The gradient is the derivative for all weights simultaneously.
1# Function with 2 variables: f(x, y) = x^2 + y^2
2def f(x, y):
3 return x**2 + y**2
4
5# Gradient: [∂f/∂x, ∂f/∂y] = [2x, 2y]
6def gradient(x, y):
7 df_dx = 2 * x # Partial derivative with respect to x
8 df_dy = 2 * y # Partial derivative with respect to y
9 return np.array([df_dx, df_dy])
10
11# At point (3, 4)
12x, y = 3, 4
13grad = gradient(x, y)
14print(f"Gradient at ({x}, {y}): {grad}") # [6, 8]
15
16# Gradient points in the direction of steepest increase
17# To minimize f, move in the opposite direction: -gradientThis is how neural networks learn!
1# Goal: Find minimum of f(x) = x^2 - 4x + 4
2def f(x):
3 return x**2 - 4*x + 4
4
5def f_derivative(x):
6 return 2*x - 4
7
8# Start at random position
9x = 10.0
10learning_rate = 0.1
11history = [x]
12
13print("Gradient Descent Optimization:")
14print(f"Step 0: x = {x:.4f}, f(x) = {f(x):.4f}")
15
16# Gradient descent loop
17for step in range(1, 11):
18 # Calculate gradient (slope)
19 grad = f_derivative(x)
20
21 # Update position (move opposite to gradient)
22 x = x - learning_rate * grad
23 history.append(x)
24
25 print(f"Step {step}: x = {x:.4f}, f(x) = {f(x):.4f}")
26
27print(f"\nConverged to x = {x:.4f} (true minimum: x = 2)")What Just Happened?
This is exactly how neural networks adjust weights during training!
The most important algorithm in deep learning uses the chain rule of calculus to compute gradients efficiently.
1# Simple neural network: y = relu(Wx + b)
2# where relu(z) = max(0, z)
3
4# Forward pass
5x = 2.0 # Input
6W = 3.0 # Weight
7b = 1.0 # Bias
8z = W * x + b # z = 7.0
9y = max(0, z) # y = 7.0 (relu activation)
10
11print(f"Forward: x={x} → z={z} → y={y}")
12
13# Backward pass (backpropagation)
14# Given: gradient of loss with respect to output
15dL_dy = 1.0 # Assume loss gradient is 1
16
17# Chain rule: dL/dW = dL/dy * dy/dz * dz/dW
18dy_dz = 1 if z > 0 else 0 # relu derivative
19dz_dW = x # derivative of (Wx+b) with respect to W
20
21dL_dW = dL_dy * dy_dz * dz_dW
22print(f"Backward: dL/dW = {dL_dW}")
23
24# Weight update
25learning_rate = 0.01
26W_new = W - learning_rate * dL_dW
27print(f"Weight update: {W} → {W_new}")Key Insight: Backpropagation computes gradients for all weights by propagating errors backward through the network using the chain rule.
Without calculus, neural networks couldn't learn!
AI must handle uncertainty and make decisions with incomplete information. Probability is the mathematical framework for reasoning under uncertainty.
1import numpy as np
2
3# Rolling a fair dice
4outcomes = np.array([1, 2, 3, 4, 5, 6])
5probabilities = np.array([1/6, 1/6, 1/6, 1/6, 1/6, 1/6])
6
7print("Dice outcomes and probabilities:")
8for outcome, prob in zip(outcomes, probabilities):
9 print(f" P({outcome}) = {prob:.3f}")
10
11# Expected value: average outcome
12expected_value = np.sum(outcomes * probabilities)
13print(f"\nExpected value: {expected_value:.2f}")
14# Meaning: On average, we expect to roll 3.5Normal Distribution (Gaussian)
Most common in AI—used for initializing weights, modeling noise:
1# Generate samples from normal distribution
2mean = 0
3std_dev = 1
4samples = np.random.normal(mean, std_dev, 1000)
5
6print(f"Mean: {np.mean(samples):.3f}")
7print(f"Std Dev: {np.std(samples):.3f}")
8
9# Weight initialization in neural networks
10def initialize_weights(shape):
11 """Initialize weights from normal distribution"""
12 return np.random.normal(0, 0.1, shape)
13
14weights = initialize_weights((100, 50))
15print(f"\nWeight matrix shape: {weights.shape}")
16print(f"Weight mean: {np.mean(weights):.4f}")
17print(f"Weight std: {np.std(weights):.4f}")Bernoulli Distribution (Binary Events)
Used in binary classification (yes/no, spam/not spam):
1# Coin flip: P(heads) = 0.7
2p_heads = 0.7
3flips = np.random.binomial(1, p_heads, 100)
4
5print(f"Probability of heads: {p_heads}")
6print(f"Actual heads in 100 flips: {np.sum(flips)}")
7print(f"Empirical probability: {np.mean(flips):.2f}")Bayes' Theorem is fundamental to probabilistic AI:
1P(A|B) = P(B|A) * P(A) / P(B)Example: Medical Diagnosis
1# Disease detection
2# P(Disease) = 0.01 (1% of population has disease)
3# P(Positive|Disease) = 0.99 (test correctly detects disease)
4# P(Positive|No Disease) = 0.05 (5% false positive rate)
5
6# Question: If test is positive, what's probability of having disease?
7
8P_disease = 0.01
9P_positive_given_disease = 0.99
10P_positive_given_no_disease = 0.05
11P_no_disease = 1 - P_disease
12
13# Total probability of positive test
14P_positive = (P_positive_given_disease * P_disease +
15 P_positive_given_no_disease * P_no_disease)
16
17# Bayes' theorem
18P_disease_given_positive = (P_positive_given_disease * P_disease) / P_positive
19
20print("Medical Test Analysis:")
21print(f"P(Disease) = {P_disease:.2%}")
22print(f"P(Positive|Disease) = {P_positive_given_disease:.2%}")
23print(f"P(Positive) = {P_positive:.2%}")
24print(f"\nP(Disease|Positive) = {P_disease_given_positive:.2%}")
25print("\nSurprisingly low! Even with positive test, only 16.6% chance.")
26print("This is because the disease is rare (1% prevalence).")Mean, Variance, Standard Deviation
1# Neural network activation statistics
2activations = np.random.randn(1000) * 2 + 5
3
4mean = np.mean(activations)
5variance = np.var(activations)
6std_dev = np.std(activations)
7
8print("Neuron Activation Statistics:")
9print(f" Mean: {mean:.3f}")
10print(f" Variance: {variance:.3f}")
11print(f" Std Dev: {std_dev:.3f}")
12
13# Why it matters: Batch normalization uses these to stabilize trainingHow AI learns probability distributions from data:
1# Observed data: coin flips
2data = np.array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1]) # 1=heads, 0=tails
3
4# Maximum likelihood estimate: probability that maximizes likelihood
5p_heads_mle = np.mean(data)
6
7print(f"Observed flips: {data}")
8print(f"Maximum Likelihood Estimate: P(heads) = {p_heads_mle:.2f}")
9print("\nMLE says: 80% chance of heads (8/10 observed)")Key Insight: Modern AI outputs probabilities, not certainties. Understanding probability is essential for interpretable AI.
Now let's see how all three mathematical concepts work together in a real neural network.
1import numpy as np
2
3# ============= NETWORK ARCHITECTURE =============
4# Input: 4 features
5# Hidden layer: 3 neurons
6# Output: 2 classes
7
8# Linear Algebra: Initialize weights and biases
9np.random.seed(42)
10W1 = np.random.randn(4, 3) * 0.1 # Input → Hidden
11b1 = np.zeros(3)
12
13W2 = np.random.randn(3, 2) * 0.1 # Hidden → Output
14b2 = np.zeros(2)
15
16print("Neural Network Architecture:")
17print(f"Input → Hidden: W1 shape {W1.shape}")
18print(f"Hidden → Output: W2 shape {W2.shape}")
19print()
20
21# Sample input
22X = np.array([1.0, 2.0, 3.0, 4.0])
23print(f"Input: {X}")
24print()
25
26# ============= FORWARD PASS =============
27
28# Hidden layer
29z1 = np.dot(X, W1) + b1 # Linear algebra: matrix multiplication
30print(f"Hidden layer pre-activation (z1): {z1}")
31
32# ReLU activation (calculus: piecewise linear function)
33a1 = np.maximum(0, z1)
34print(f"Hidden layer activations (a1): {a1}")
35print()
36
37# Output layer
38z2 = np.dot(a1, W2) + b2
39print(f"Output pre-activation (z2): {z2}")
40
41# Softmax activation (probability: convert to probabilities)
42exp_z2 = np.exp(z2 - np.max(z2)) # Numerical stability
43probabilities = exp_z2 / np.sum(exp_z2)
44
45print(f"Output probabilities: {probabilities}")
46print(f" Class 0: {probabilities[0]:.2%}")
47print(f" Class 1: {probabilities[1]:.2%}")
48print(f"Sum of probabilities: {np.sum(probabilities):.4f}")
49print()
50
51# ============= BACKWARD PASS (Simplified) =============
52
53# True label: class 1
54y_true = 1
55
56# Loss: Cross-entropy (probability-based loss)
57loss = -np.log(probabilities[y_true])
58print(f"Loss: {loss:.4f}")
59print()
60
61# Gradients (calculus: chain rule)
62dL_dz2 = probabilities.copy()
63dL_dz2[y_true] -= 1 # Gradient of cross-entropy + softmax
64
65print(f"Output gradient: {dL_dz2}")
66
67# Backpropagate to hidden layer
68dL_dW2 = np.outer(a1, dL_dz2)
69dL_da1 = np.dot(dL_dz2, W2.T)
70
71print(f"Hidden layer gradient shape: {dL_da1.shape}")
72print()
73
74# ============= WEIGHT UPDATE =============
75
76learning_rate = 0.01
77W2_new = W2 - learning_rate * dL_dW2
78
79print(f"Weight updated (gradient descent)")
80print(f"Max weight change: {np.max(np.abs(W2_new - W2)):.6f}")
81print()
82
83print("=" * 60)
84print("MATH CONCEPTS APPLIED:")
85print("=" * 60)
86print("✓ Linear Algebra: Matrix multiplication (forward pass)")
87print("✓ Calculus: Gradients & backpropagation (learning)")
88print("✓ Probability: Softmax probabilities & cross-entropy")
89print("=" * 60)1. Linear Algebra = Data Representation
2. Calculus = Learning Algorithm
3. Probability = Decision Making
Together: These three concepts enable neural networks to learn from data and make intelligent predictions!
Implement a complete neural network using only NumPy, demonstrating every mathematical concept we learned. This is how frameworks like TensorFlow work under the hood!
1# Neural Network from Scratch - Pure Mathematics!
2# Implementing forward pass, backpropagation, and training
3
4import numpy as np
5import matplotlib.pyplot as plt
6
7print("=" * 70)
8print("NEURAL NETWORK FROM MATHEMATICAL SCRATCH")
9print("=" * 70)
10print()
11
12# ============================================================================
13# DATASET: XOR Problem (Non-linearly Separable)
14# ============================================================================
15
16# XOR inputs and outputs
17X = np.array([
18 [0, 0],
19 [0, 1],
20 [1, 0],
21 [1, 1]
22])
23
24y = np.array([[0], [1], [1], [0]]) # XOR output
25
26print("XOR Problem:")
27print("Input → Output")
28print("-" * 20)
29for i in range(len(X)):
30 print(f"{X[i]} → {y[i][0]}")
31print()
32
33# ============================================================================
34# NEURAL NETWORK ARCHITECTURE
35# ============================================================================
36
37# Network: 2 inputs → 4 hidden neurons → 1 output
38input_size = 2
39hidden_size = 4
40output_size = 1
41
42# Initialize weights (Linear Algebra + Probability)
43np.random.seed(42)
44W1 = np.random.randn(input_size, hidden_size) * 0.5
45b1 = np.zeros((1, hidden_size))
46
47W2 = np.random.randn(hidden_size, output_size) * 0.5
48b2 = np.zeros((1, output_size))
49
50print("Network Architecture:")
51print(f" Input layer: {input_size} neurons")
52print(f" Hidden layer: {hidden_size} neurons")
53print(f" Output layer: {output_size} neuron")
54print()
55print(f" W1 shape: {W1.shape} (weights input→hidden)")
56print(f" W2 shape: {W2.shape} (weights hidden→output)")
57print()
58
59# ============================================================================
60# ACTIVATION FUNCTIONS (Calculus)
61# ============================================================================
62
63def sigmoid(x):
64 """Sigmoid activation: σ(x) = 1 / (1 + e^(-x))"""
65 return 1 / (1 + np.exp(-np.clip(x, -500, 500)))
66
67def sigmoid_derivative(x):
68 """Derivative: σ'(x) = σ(x) * (1 - σ(x))"""
69 return x * (1 - x)
70
71# ============================================================================
72# FORWARD PASS (Linear Algebra)
73# ============================================================================
74
75def forward_pass(X, W1, b1, W2, b2):
76 """
77 Forward propagation through the network
78 Returns: hidden activations, output, predictions
79 """
80 # Hidden layer
81 z1 = np.dot(X, W1) + b1 # Linear transformation
82 a1 = sigmoid(z1) # Activation
83
84 # Output layer
85 z2 = np.dot(a1, W2) + b2 # Linear transformation
86 a2 = sigmoid(z2) # Activation
87
88 return a1, a2
89
90# ============================================================================
91# LOSS FUNCTION (Probability)
92# ============================================================================
93
94def compute_loss(y_true, y_pred):
95 """
96 Binary cross-entropy loss
97 L = -[y*log(ŷ) + (1-y)*log(1-ŷ)]
98 """
99 m = y_true.shape[0]
100 loss = -np.mean(y_true * np.log(y_pred + 1e-8) +
101 (1 - y_true) * np.log(1 - y_pred + 1e-8))
102 return loss
103
104# ============================================================================
105# BACKPROPAGATION (Calculus: Chain Rule)
106# ============================================================================
107
108def backward_pass(X, y, a1, a2, W2):
109 """
110 Backpropagation: compute gradients using chain rule
111 Returns: gradients for W1, b1, W2, b2
112 """
113 m = X.shape[0]
114
115 # Output layer gradients
116 dz2 = a2 - y # dL/dz2
117 dW2 = np.dot(a1.T, dz2) / m # dL/dW2
118 db2 = np.sum(dz2, axis=0, keepdims=True) / m # dL/db2
119
120 # Hidden layer gradients (chain rule!)
121 dz1 = np.dot(dz2, W2.T) * sigmoid_derivative(a1)
122 dW1 = np.dot(X.T, dz1) / m
123 db1 = np.sum(dz1, axis=0, keepdims=True) / m
124
125 return dW1, db1, dW2, db2
126
127# ============================================================================
128# TRAINING LOOP (Gradient Descent)
129# ============================================================================
130
131learning_rate = 1.0
132epochs = 10000
133loss_history = []
134
135print("Training Neural Network...")
136print("-" * 70)
137
138for epoch in range(epochs):
139 # Forward pass
140 a1, a2 = forward_pass(X, W1, b1, W2, b2)
141
142 # Compute loss
143 loss = compute_loss(y, a2)
144 loss_history.append(loss)
145
146 # Backward pass
147 dW1, db1, dW2, db2 = backward_pass(X, y, a1, a2, W2)
148
149 # Update weights (gradient descent)
150 W1 -= learning_rate * dW1
151 b1 -= learning_rate * db1
152 W2 -= learning_rate * dW2
153 b2 -= learning_rate * db2
154
155 # Print progress
156 if (epoch + 1) % 1000 == 0:
157 accuracy = np.mean((a2 > 0.5) == y) * 100
158 print(f"Epoch {epoch+1:5d} | Loss: {loss:.6f} | Accuracy: {accuracy:.2f}%")
159
160print()
161print("Training Complete!")
162print("=" * 70)
163print()
164
165# ============================================================================
166# EVALUATION
167# ============================================================================
168
169# Final predictions
170a1_final, predictions = forward_pass(X, W1, b1, W2, b2)
171
172print("Final Predictions:")
173print("-" * 40)
174print("Input | Prediction | Truth | Correct?")
175print("-" * 40)
176for i in range(len(X)):
177 pred = predictions[i][0]
178 truth = y[i][0]
179 correct = "✓" if (pred > 0.5) == truth else "✗"
180 print(f"{X[i]} | {pred:.6f} | {truth} | {correct}")
181print()
182
183final_accuracy = np.mean((predictions > 0.5) == y) * 100
184print(f"Final Accuracy: {final_accuracy:.2f}%")
185print()
186
187# ============================================================================
188# VISUALIZATIONS
189# ============================================================================
190
191# Plot 1: Loss over time
192fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18, 5))
193fig.suptitle('Neural Network Training - Mathematical Analysis',
194 fontsize=16, fontweight='bold')
195
196ax1.plot(loss_history, linewidth=2, color='red')
197ax1.set_xlabel('Epoch', fontweight='bold')
198ax1.set_ylabel('Loss (Cross-Entropy)', fontweight='bold')
199ax1.set_title('Learning Curve: Loss Decreases via Gradient Descent',
200 fontweight='bold')
201ax1.grid(True, alpha=0.3)
202
203# Plot 2: Decision boundary
204xx, yy = np.meshgrid(np.linspace(-0.5, 1.5, 100),
205 np.linspace(-0.5, 1.5, 100))
206Z = []
207for i in range(len(xx)):
208 row = []
209 for j in range(len(xx[0])):
210 point = np.array([[xx[i][j], yy[i][j]]])
211 _, pred = forward_pass(point, W1, b1, W2, b2)
212 row.append(pred[0][0])
213 Z.append(row)
214Z = np.array(Z)
215
216contour = ax2.contourf(xx, yy, Z, levels=20, cmap='RdYlGn', alpha=0.6)
217ax2.scatter(X[:, 0], X[:, 1], c=y.ravel(), cmap='RdYlGn',
218 s=200, edgecolors='black', linewidths=2)
219ax2.set_xlabel('Input X1', fontweight='bold')
220ax2.set_ylabel('Input X2', fontweight='bold')
221ax2.set_title('Decision Boundary Learned by Neural Network',
222 fontweight='bold')
223plt.colorbar(contour, ax=ax2, label='Prediction')
224
225# Plot 3: Network architecture
226ax3.axis('off')
227ax3.text(0.5, 0.9, 'Neural Network Architecture',
228 ha='center', fontsize=14, fontweight='bold')
229
230layers_text = """
231Layer 1 (Input): 2 neurons
232 ↓ W1 [2×4] + b1
233Layer 2 (Hidden): 4 neurons + Sigmoid
234 ↓ W2 [4×1] + b2
235Layer 3 (Output): 1 neuron + Sigmoid
236
237Mathematical Operations:
238━━━━━━━━━━━━━━━━━━━━━━━━━
239Forward Pass:
240 z1 = X @ W1 + b1
241 a1 = sigmoid(z1)
242 z2 = a1 @ W2 + b2
243 a2 = sigmoid(z2)
244
245Backward Pass:
246 dL/dW2 = a1.T @ (a2 - y)
247 dL/dW1 = X.T @ (dL/da1)
248
249Weight Update:
250 W ← W - α * dL/dW
251━━━━━━━━━━━━━━━━━━━━━━━━━
252
253Total Parameters: 17
254(2×4 + 4 + 4×1 + 1)
255"""
256
257ax3.text(0.1, 0.7, layers_text, ha='left', va='top',
258 fontsize=10, family='monospace',
259 bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))
260
261plt.tight_layout()
262plt.show()
263
264# ============================================================================
265# MATHEMATICAL CONCEPTS DEMONSTRATED
266# ============================================================================
267
268print("=" * 70)
269print("MATHEMATICAL CONCEPTS YOU JUST IMPLEMENTED")
270print("=" * 70)
271print()
272print("1️⃣ LINEAR ALGEBRA")
273print(" ✓ Matrix multiplication (np.dot)")
274print(" ✓ Transpose operations (X.T, a1.T)")
275print(" ✓ Broadcasting (adding bias vectors)")
276print()
277print("2️⃣ CALCULUS")
278print(" ✓ Sigmoid derivative: σ'(x) = σ(x)(1-σ(x))")
279print(" ✓ Chain rule in backpropagation")
280print(" ✓ Gradient descent: W ← W - α∇L")
281print()
282print("3️⃣ PROBABILITY")
283print(" ✓ Sigmoid outputs probabilities")
284print(" ✓ Cross-entropy loss function")
285print(" ✓ Maximum likelihood estimation")
286print()
287print("=" * 70)
288print("🎉 You built a neural network using pure mathematics!")
289print("=" * 70)This comprehensive example demonstrates every mathematical concept we learned by building a complete neural network from scratch using only NumPy.
1. Solves the XOR Problem
XOR is a classic problem that's not linearly separable—it requires a neural network with hidden layers. Linear models (like logistic regression) can't solve it, but our neural network achieves 100% accuracy!
2. Implements Core Mathematical Operations
Linear Algebra:
np.dot(X, W1): Matrix multiplication for forward passX.T: Transpose for gradient computationCalculus:
W -= learning_rate * dWProbability:
1. Understanding > Black Boxes
By implementing from scratch, you see:
2. Real Framework Foundations
This is essentially how TensorFlow and PyTorch work! They:
3. Debugging Deep Learning
Understanding the math helps you:
Forward Pass = Function Composition
1f(x) = σ(W₂ · σ(W₁·x + b₁) + b₂)Backward Pass = Chain Rule
1∂L/∂W₁ = ∂L/∂a₂ · ∂a₂/∂z₂ · ∂z₂/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂W₁Learning = Gradient Descent
1W(t+1) = W(t) - α · ∇L(W)1. Loss Curve: Shows gradient descent working—loss decreases smoothly
2. Decision Boundary: Neural network learned to separate XOR classes
3. Architecture Diagram: See how data flows through layers
This 200-line example encapsulates everything you need to understand about neural network mathematics!
Outstanding work! You now understand the mathematical foundations of AI. You've seen how linear algebra, calculus, and probability work together to enable neural networks to learn.
Next: In Lesson 3: Introduction to TensorFlow, we'll use these mathematical concepts to build neural networks with Google's powerful deep learning framework. You'll learn:
No more manual backpropagation—TensorFlow automates everything we just coded by hand!
Your AI Journey Continues → Ready to harness the power of modern deep learning frameworks? Let's go! 🚀
Your progress is saved in this browser, on this device only.