Build neural networks from scratch, understanding perceptrons, activation functions, forward propagation, and the mathematical foundations of deep learning.
Neural networks are the foundation of modern deep learning, powering everything from image recognition to natural language processing. Understanding how neural networks work at a fundamental level is essential for building, debugging, and optimizing deep learning systems.
While frameworks like PyTorch and TensorFlow make building neural networks easy, knowing the underlying mathematics and implementing networks from scratch provides invaluable intuition. This knowledge enables you to diagnose problems, design custom architectures, and push the boundaries of what's possible with deep learning.
In this comprehensive lesson, you'll build neural networks from the ground up using NumPy. You'll understand perceptrons, activation functions, forward propagation, loss computation, and the complete architecture of feedforward neural networks. This foundation will prepare you for advanced topics like backpropagation, CNNs, and Transformers.
The perceptron is the simplest neural network unit, inspired by biological neurons. Understanding perceptrons is essential for grasping how complex deep learning models work.
Biological Inspiration:
Biological neurons receive signals through dendrites, process them in the cell body, and send output through axons. The perceptron mimics this:
Mathematical Model:
A perceptron computes a weighted sum of inputs plus bias, then applies an activation function:
1z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b = Σ(wᵢxᵢ) + b
2y = activation(z)Implementing a Perceptron:
1import numpy as np
2
3class Perceptron:
4 """Single perceptron unit"""
5
6 def __init__(self, n_inputs):
7 # Initialize weights randomly
8 self.weights = np.random.randn(n_inputs) * 0.01
9 self.bias = 0.0
10
11 def forward(self, x):
12 """Forward pass through perceptron"""
13 # Weighted sum
14 z = np.dot(x, self.weights) + self.bias
15 return z
16
17 def step_activation(self, z):
18 """Binary step activation (original perceptron)"""
19 return 1 if z >= 0 else 0
20
21 def predict(self, x):
22 """Make prediction"""
23 z = self.forward(x)
24 return self.step_activation(z)
25
26# Example: AND gate with perceptron
27perceptron = Perceptron(n_inputs=2)
28perceptron.weights = np.array([0.5, 0.5]) # Manually set for AND gate
29perceptron.bias = -0.7
30
31# Test AND gate
32X_test = np.array([
33 [0, 0],
34 [0, 1],
35 [1, 0],
36 [1, 1]
37])
38
39print("AND Gate with Perceptron:")
40for x in X_test:
41 prediction = perceptron.predict(x)
42 print(f"Input: {x} → Output: {prediction}")Perceptron Limitations:
The perceptron can only learn linearly separable patterns (patterns that can be separated by a straight line/hyperplane). Famous example: XOR problem cannot be solved by a single perceptron.
XOR Problem:
1Input Output
20 0 → 0
30 1 → 1
41 0 → 1
51 1 → 0No single line can separate the 1s from 0s!
Solution: Multi-Layer Perceptrons
By stacking multiple layers of perceptrons with non-linear activations, we can learn complex, non-linear patterns. This is the foundation of deep learning.
Key Insights:
Activation functions introduce non-linearity into neural networks, enabling them to learn complex patterns. Without activation functions, even deep networks would be equivalent to single-layer linear models.
Why Non-Linearity Matters:
Consider a network without activation functions:
1Layer 1: z₁ = W₁x + b₁
2Layer 2: z₂ = W₂z₁ + b₂ = W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂)This simplifies to a single linear transformation! Activation functions break this linearity.
Common Activation Functions:
1. Sigmoid (Logistic):
1σ(z) = 1 / (1 + e^(-z))
2Range: (0, 1)
3Use case: Binary classification output layerPros:
Cons:
2. Hyperbolic Tangent (tanh):
1tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))
2Range: (-1, 1)
3Use case: Hidden layers (better than sigmoid)Pros:
Cons:
3. ReLU (Rectified Linear Unit):
1ReLU(z) = max(0, z)
2Range: [0, ∞)
3Use case: Default choice for hidden layersPros:
Cons:
Implementing Activation Functions:
1import numpy as np
2import matplotlib.pyplot as plt
3
4class ActivationFunctions:
5 """Common activation functions"""
6
7 @staticmethod
8 def sigmoid(z):
9 """Sigmoid activation"""
10 return 1 / (1 + np.exp(-np.clip(z, -500, 500))) # Clip for stability
11
12 @staticmethod
13 def tanh(z):
14 """Hyperbolic tangent"""
15 return np.tanh(z)
16
17 @staticmethod
18 def relu(z):
19 """ReLU activation"""
20 return np.maximum(0, z)
21
22 @staticmethod
23 def leaky_relu(z, alpha=0.01):
24 """Leaky ReLU (fixes dying ReLU)"""
25 return np.where(z > 0, z, alpha * z)
26
27 @staticmethod
28 def softmax(z):
29 """Softmax for multi-class classification"""
30 exp_z = np.exp(z - np.max(z, axis=-1, keepdims=True)) # Stability
31 return exp_z / np.sum(exp_z, axis=-1, keepdims=True)
32
33# Visualize activations
34z = np.linspace(-5, 5, 100)
35
36plt.figure(figsize=(12, 4))
37
38plt.subplot(1, 3, 1)
39plt.plot(z, ActivationFunctions.sigmoid(z), label='Sigmoid', linewidth=2)
40plt.plot(z, ActivationFunctions.tanh(z), label='Tanh', linewidth=2)
41plt.grid(True, alpha=0.3)
42plt.legend()
43plt.title('Sigmoid vs Tanh')
44plt.xlabel('z')
45plt.ylabel('activation(z)')
46
47plt.subplot(1, 3, 2)
48plt.plot(z, ActivationFunctions.relu(z), label='ReLU', linewidth=2)
49plt.plot(z, ActivationFunctions.leaky_relu(z), label='Leaky ReLU', linewidth=2)
50plt.grid(True, alpha=0.3)
51plt.legend()
52plt.title('ReLU Variants')
53plt.xlabel('z')
54plt.ylabel('activation(z)')
55
56plt.subplot(1, 3, 3)
57# Softmax example
58z_soft = np.array([[1, 2, 3], [1, 1, 1], [3, 2, 1]])
59softmax_out = ActivationFunctions.softmax(z_soft)
60plt.bar(range(3), softmax_out[0], alpha=0.7, label='Input [1,2,3]')
61plt.bar(range(3), softmax_out[1], alpha=0.7, label='Input [1,1,1]')
62plt.title('Softmax Example')
63plt.xlabel('Class')
64plt.ylabel('Probability')
65plt.legend()
66
67plt.tight_layout()
68plt.show()Choosing Activation Functions:
Hidden Layers:
Output Layer:
Modern Variations:
Forward propagation is the process of passing input through network layers to compute output. Understanding this process is essential for building and debugging neural networks.
Multi-Layer Perceptron (MLP) Architecture:
A typical MLP has:
Mathematical Formulation:
For a 2-hidden-layer network:
1Input: x ∈ ℝⁿ
2
3Layer 1:
4 z₁ = W₁x + b₁ (linear transformation)
5 a₁ = activation(z₁) (non-linear activation)
6
7Layer 2:
8 z₂ = W₂a₁ + b₂
9 a₂ = activation(z₂)
10
11Output:
12 z₃ = W₃a₂ + b₃
13 ŷ = activation(z₃) (depends on task)Notation:
Implementing Forward Propagation:
1import numpy as np
2
3class NeuralNetwork:
4 """Simple feedforward neural network"""
5
6 def __init__(self, layer_sizes):
7 """
8 Args:
9 layer_sizes: List of layer sizes [input, hidden1, hidden2, ..., output]
10 """
11 self.layer_sizes = layer_sizes
12 self.num_layers = len(layer_sizes)
13 self.weights = []
14 self.biases = []
15
16 # Initialize weights and biases
17 for i in range(1, self.num_layers):
18 w = np.random.randn(layer_sizes[i], layer_sizes[i-1]) * 0.01
19 b = np.zeros((layer_sizes[i], 1))
20 self.weights.append(w)
21 self.biases.append(b)
22
23 def relu(self, z):
24 """ReLU activation"""
25 return np.maximum(0, z)
26
27 def sigmoid(self, z):
28 """Sigmoid activation"""
29 return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
30
31 def forward(self, X):
32 """
33 Forward propagation
34
35 Args:
36 X: Input data (n_features, n_samples)
37
38 Returns:
39 Output, cache of intermediate values
40 """
41 cache = {'A0': X} # Store activations
42 A = X
43
44 # Forward through hidden layers (use ReLU)
45 for i in range(len(self.weights) - 1):
46 Z = np.dot(self.weights[i], A) + self.biases[i]
47 A = self.relu(Z)
48 cache[f'Z{i+1}'] = Z
49 cache[f'A{i+1}'] = A
50
51 # Output layer (use sigmoid for binary classification)
52 Z_out = np.dot(self.weights[-1], A) + self.biases[-1]
53 A_out = self.sigmoid(Z_out)
54 cache[f'Z{len(self.weights)}'] = Z_out
55 cache[f'A{len(self.weights)}'] = A_out
56
57 return A_out, cache
58
59# Example: Create and test network
60nn = NeuralNetwork(layer_sizes=[4, 8, 8, 1]) # 4 input, 2 hidden (8 neurons each), 1 output
61
62# Forward pass with random input
63X = np.random.randn(4, 10) # 4 features, 10 samples
64output, cache = nn.forward(X)
65
66print(f"Input shape: {X.shape}")
67print(f"Output shape: {output.shape}")
68print(f"Output (first 5 samples): {output[:, :5].flatten()}")Vectorized Implementation:
Vectorization processes multiple samples simultaneously:
1# Instead of looping over samples
2for i in range(n_samples):
3 z = np.dot(W, X[:, i]) + b # Slow
4
5# Vectorize
6Z = np.dot(W, X) + b # Fast! Processes all samples at onceMatrix Shapes:
1X: (n_features, n_samples)
2W: (n_neurons, n_features)
3b: (n_neurons, 1)
4Z: (n_neurons, n_samples)
5A: (n_neurons, n_samples)Broadcasting:
1Z = np.dot(W, X) + b
2# b: (n_neurons, 1) is broadcast to (n_neurons, n_samples)Key Concepts:
Loss functions quantify how well neural network predictions match true labels. Choosing the right loss function is crucial for effective training.
Purpose of Loss Functions:
The loss function L(ŷ, y) measures the discrepancy between:
Goal: Minimize loss through training (gradient descent)
Common Loss Functions:
1. Binary Cross-Entropy (Log Loss)
For binary classification (2 classes):
1L(ŷ, y) = -[y log(ŷ) + (1-y) log(1-ŷ)]
2
3For m samples:
4J = (1/m) Σᵢ [-yᵢ log(ŷᵢ) - (1-yᵢ) log(1-ŷᵢ)]When to use: Binary classification with sigmoid output
Properties:
2. Categorical Cross-Entropy
For multi-class classification (K classes):
1L(ŷ, y) = -Σₖ yₖ log(ŷₖ)
2
3where y is one-hot encodedWhen to use: Multi-class classification with softmax output
3. Mean Squared Error (MSE)
For regression tasks:
1MSE = (1/m) Σᵢ (ŷᵢ - yᵢ)²When to use: Regression problems
Properties:
4. Mean Absolute Error (MAE)
Alternative for regression:
1MAE = (1/m) Σᵢ |ŷᵢ - yᵢ|Properties:
Implementing Loss Functions:
1import numpy as np
2
3class LossFunctions:
4 """Common loss functions"""
5
6 @staticmethod
7 def binary_cross_entropy(y_pred, y_true):
8 """
9 Binary cross-entropy loss
10
11 Args:
12 y_pred: Predictions (0-1), shape (n_samples,)
13 y_true: True labels (0 or 1), shape (n_samples,)
14
15 Returns:
16 Average loss
17 """
18 m = y_true.shape[0]
19 # Clip predictions to avoid log(0)
20 y_pred = np.clip(y_pred, 1e-7, 1 - 1e-7)
21
22 loss = -np.mean(
23 y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred)
24 )
25 return loss
26
27 @staticmethod
28 def categorical_cross_entropy(y_pred, y_true):
29 """
30 Categorical cross-entropy loss
31
32 Args:
33 y_pred: Predictions (probabilities), shape (n_samples, n_classes)
34 y_true: True labels (one-hot), shape (n_samples, n_classes)
35
36 Returns:
37 Average loss
38 """
39 m = y_true.shape[0]
40 y_pred = np.clip(y_pred, 1e-7, 1 - 1e-7)
41
42 loss = -np.mean(np.sum(y_true * np.log(y_pred), axis=1))
43 return loss
44
45 @staticmethod
46 def mse(y_pred, y_true):
47 """
48 Mean squared error
49
50 Args:
51 y_pred: Predictions, shape (n_samples,)
52 y_true: True values, shape (n_samples,)
53
54 Returns:
55 Average loss
56 """
57 return np.mean((y_pred - y_true) ** 2)
58
59 @staticmethod
60 def mae(y_pred, y_true):
61 """Mean absolute error"""
62 return np.mean(np.abs(y_pred - y_true))
63
64# Example: Compute losses
65np.random.seed(42)
66
67# Binary classification
68y_true_binary = np.array([1, 0, 1, 1, 0])
69y_pred_binary = np.array([0.9, 0.1, 0.8, 0.6, 0.2])
70
71bce_loss = LossFunctions.binary_cross_entropy(y_pred_binary, y_true_binary)
72print(f"Binary Cross-Entropy Loss: {bce_loss:.4f}")
73
74# Multi-class classification
75y_true_multi = np.array([
76 [1, 0, 0], # Class 0
77 [0, 1, 0], # Class 1
78 [0, 0, 1] # Class 2
79])
80y_pred_multi = np.array([
81 [0.7, 0.2, 0.1],
82 [0.1, 0.8, 0.1],
83 [0.2, 0.2, 0.6]
84])
85
86cce_loss = LossFunctions.categorical_cross_entropy(y_pred_multi, y_true_multi)
87print(f"Categorical Cross-Entropy Loss: {cce_loss:.4f}")
88
89# Regression
90y_true_reg = np.array([1.5, 2.0, 3.0, 4.5, 5.0])
91y_pred_reg = np.array([1.6, 2.1, 2.8, 4.4, 5.2])
92
93mse_loss = LossFunctions.mse(y_pred_reg, y_true_reg)
94mae_loss = LossFunctions.mae(y_pred_reg, y_true_reg)
95
96print(f"MSE Loss: {mse_loss:.4f}")
97print(f"MAE Loss: {mae_loss:.4f}")Regularization Terms:
Often add regularization to loss to prevent overfitting:
L2 Regularization (Weight Decay):
1J = Loss + (λ/2m) Σᵢ ||Wᵢ||²L1 Regularization:
1J = Loss + (λ/m) Σᵢ |Wᵢ|Choosing Loss Functions:
| Task | Output Activation | Loss Function |
|------|------------------|---------------|
| Binary Classification | Sigmoid | Binary Cross-Entropy |
| Multi-class Classification | Softmax | Categorical Cross-Entropy |
| Regression | Linear | MSE or MAE |
| Multi-label Classification | Sigmoid | Binary Cross-Entropy (per label) |
Key Insights:
Now we combine all components to build complete, trainable neural networks from scratch using NumPy. This provides deep understanding before using frameworks.
Complete Neural Network Class:
1import numpy as np
2import matplotlib.pyplot as plt
3
4class CompleteNeuralNetwork:
5 """Complete feedforward neural network with training"""
6
7 def __init__(self, layer_sizes, learning_rate=0.01):
8 """
9 Initialize network
10
11 Args:
12 layer_sizes: List [input_size, hidden1, hidden2, ..., output]
13 learning_rate: Learning rate for gradient descent
14 """
15 self.layer_sizes = layer_sizes
16 self.num_layers = len(layer_sizes)
17 self.learning_rate = learning_rate
18
19 # Initialize parameters
20 self.parameters = {}
21 for i in range(1, self.num_layers):
22 # He initialization for ReLU
23 self.parameters[f'W{i}'] = np.random.randn(
24 layer_sizes[i], layer_sizes[i-1]
25 ) * np.sqrt(2.0 / layer_sizes[i-1])
26 self.parameters[f'b{i}'] = np.zeros((layer_sizes[i], 1))
27
28 def relu(self, Z):
29 """ReLU activation"""
30 return np.maximum(0, Z)
31
32 def relu_derivative(self, Z):
33 """ReLU derivative"""
34 return (Z > 0).astype(float)
35
36 def sigmoid(self, Z):
37 """Sigmoid activation"""
38 return 1 / (1 + np.exp(-np.clip(Z, -500, 500)))
39
40 def sigmoid_derivative(self, A):
41 """Sigmoid derivative"""
42 return A * (1 - A)
43
44 def forward_propagation(self, X):
45 """
46 Forward propagation through network
47
48 Args:
49 X: Input (n_features, n_samples)
50
51 Returns:
52 Final activation, cache
53 """
54 cache = {'A0': X}
55 A = X
56
57 # Hidden layers with ReLU
58 for i in range(1, self.num_layers - 1):
59 Z = np.dot(self.parameters[f'W{i}'], A) + self.parameters[f'b{i}']
60 A = self.relu(Z)
61 cache[f'Z{i}'] = Z
62 cache[f'A{i}'] = A
63
64 # Output layer with sigmoid (binary classification)
65 Z_out = np.dot(
66 self.parameters[f'W{self.num_layers-1}'], A
67 ) + self.parameters[f'b{self.num_layers-1}']
68 A_out = self.sigmoid(Z_out)
69
70 cache[f'Z{self.num_layers-1}'] = Z_out
71 cache[f'A{self.num_layers-1}'] = A_out
72
73 return A_out, cache
74
75 def compute_loss(self, A_out, Y):
76 """Binary cross-entropy loss"""
77 m = Y.shape[1]
78 loss = -np.mean(
79 Y * np.log(A_out + 1e-8) + (1 - Y) * np.log(1 - A_out + 1e-8)
80 )
81 return loss
82
83 def predict(self, X):
84 """Make predictions"""
85 A_out, _ = self.forward_propagation(X)
86 predictions = (A_out > 0.5).astype(int)
87 return predictions
88
89 def accuracy(self, X, Y):
90 """Compute accuracy"""
91 predictions = self.predict(X)
92 return np.mean(predictions == Y)
93
94# Example: XOR Problem (non-linearly separable)
95print("=" * 60)
96print("Solving XOR Problem with Neural Network")
97print("=" * 60)
98
99# XOR dataset
100X = np.array([[0, 0, 1, 1],
101 [0, 1, 0, 1]])
102Y = np.array([[0, 1, 1, 0]]) # XOR output
103
104# Create network
105nn = CompleteNeuralNetwork(
106 layer_sizes=[2, 4, 1], # 2 inputs, 4 hidden, 1 output
107 learning_rate=0.1
108)
109
110# Forward pass before training
111A_out, cache = nn.forward_propagation(X)
112initial_loss = nn.compute_loss(A_out, Y)
113print(f"
114Initial Loss: {initial_loss:.4f}")
115print(f"Initial Predictions: {A_out.flatten()}")
116print(f"Initial Accuracy: {nn.accuracy(X, Y) * 100:.2f}%")
117
118# Note: Full training requires backpropagation (next lesson)
119print("
120(Full training with backpropagation in next lesson)")Network Architecture Decisions:
1. Number of Layers:
2. Neurons per Layer:
3. Activation Functions:
4. Weight Initialization:
Debugging Neural Networks:
1def debug_network(nn, X, Y):
2 """Debug network architecture and forward pass"""
3
4 print("Network Architecture:")
5 print("-" * 40)
6 for i in range(1, nn.num_layers):
7 W_shape = nn.parameters[f'W{i}'].shape
8 b_shape = nn.parameters[f'b{i}'].shape
9 print(f"Layer {i}: W{W_shape}, b{b_shape}")
10
11 print("
12Forward Propagation:")
13 print("-" * 40)
14 A_out, cache = nn.forward_propagation(X)
15
16 for key, value in cache.items():
17 print(f"{key}: {value.shape}")
18
19 print(f"
20Output range: [{A_out.min():.4f}, {A_out.max():.4f}]")
21 print(f"Loss: {nn.compute_loss(A_out, Y):.4f}")
22
23# Debug example
24debug_network(nn, X, Y)Best Practices:
Common Pitfalls:
Build a complete feedforward neural network from scratch using NumPy, including proper initialization, forward propagation, and loss computation for binary classification.
1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.datasets import make_moons
4from sklearn.model_selection import train_test_split
5
6print("=" * 70)
7print("COMPLETE NEURAL NETWORK FROM SCRATCH")
8print("=" * 70)
9
10# Set random seed for reproducibility
11np.random.seed(42)
12
13# ============================================================================
14# NEURAL NETWORK CLASS
15# ============================================================================
16class NeuralNetworkFromScratch:
17 """Complete neural network implementation"""
18
19 def __init__(self, layer_sizes, learning_rate=0.01, seed=42):
20 """
21 Initialize neural network
22
23 Args:
24 layer_sizes: List of layer sizes [input, hidden1, ..., output]
25 learning_rate: Learning rate for gradient descent
26 seed: Random seed for reproducibility
27 """
28 np.random.seed(seed)
29 self.layer_sizes = layer_sizes
30 self.num_layers = len(layer_sizes)
31 self.learning_rate = learning_rate
32 self.parameters = {}
33
34 # Initialize weights and biases with He initialization
35 for i in range(1, self.num_layers):
36 self.parameters[f'W{i}'] = np.random.randn(
37 layer_sizes[i], layer_sizes[i-1]
38 ) * np.sqrt(2.0 / layer_sizes[i-1]) # He initialization for ReLU
39
40 self.parameters[f'b{i}'] = np.zeros((layer_sizes[i], 1))
41
42 def relu(self, Z):
43 """ReLU activation function"""
44 return np.maximum(0, Z)
45
46 def sigmoid(self, Z):
47 """Sigmoid activation function"""
48 return 1 / (1 + np.exp(-np.clip(Z, -500, 500)))
49
50 def forward_propagation(self, X):
51 """
52 Forward propagation through the network
53
54 Args:
55 X: Input data (n_features, n_samples)
56
57 Returns:
58 A_out: Output activations
59 cache: Dictionary of intermediate values
60 """
61 cache = {'A0': X}
62 A = X
63
64 # Forward through hidden layers with ReLU
65 for i in range(1, self.num_layers - 1):
66 Z = np.dot(self.parameters[f'W{i}'], A) + self.parameters[f'b{i}']
67 A = self.relu(Z)
68 cache[f'Z{i}'] = Z
69 cache[f'A{i}'] = A
70
71 # Output layer with sigmoid
72 Z_out = np.dot(
73 self.parameters[f'W{self.num_layers-1}'], A
74 ) + self.parameters[f'b{self.num_layers-1}']
75 A_out = self.sigmoid(Z_out)
76
77 cache[f'Z{self.num_layers-1}'] = Z_out
78 cache[f'A{self.num_layers-1}'] = A_out
79
80 return A_out, cache
81
82 def compute_loss(self, A_out, Y):
83 """
84 Compute binary cross-entropy loss
85
86 Args:
87 A_out: Model predictions (1, n_samples)
88 Y: True labels (1, n_samples)
89
90 Returns:
91 loss: Average loss across samples
92 """
93 m = Y.shape[1]
94 loss = -np.mean(
95 Y * np.log(A_out + 1e-8) + (1 - Y) * np.log(1 - A_out + 1e-8)
96 )
97 return loss
98
99 def predict(self, X):
100 """Make binary predictions"""
101 A_out, _ = self.forward_propagation(X)
102 return (A_out > 0.5).astype(int)
103
104 def accuracy(self, X, Y):
105 """Compute accuracy"""
106 predictions = self.predict(X)
107 return np.mean(predictions == Y)
108
109# ============================================================================
110# GENERATE DATASET
111# ============================================================================
112print("
1131. Generating Dataset...")
114print("-" * 70)
115
116# Generate moons dataset (non-linearly separable)
117X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)
118X = X.T # Transpose to (n_features, n_samples)
119y = y.reshape(1, -1) # Reshape to (1, n_samples)
120
121# Split into train and test
122X_train, X_test, y_train, y_test = train_test_split(
123 X.T, y.T, test_size=0.2, random_state=42
124)
125X_train, X_test = X_train.T, X_test.T
126y_train, y_test = y_train.T, y_test.T
127
128print(f"Training samples: {X_train.shape[1]}")
129print(f"Test samples: {X_test.shape[1]}")
130print(f"Features: {X_train.shape[0]}")
131print(f"Class distribution: {np.bincount(y_train.flatten().astype(int))}")
132
133# ============================================================================
134# CREATE AND TEST NETWORK
135# ============================================================================
136print("
1372. Creating Neural Network...")
138print("-" * 70)
139
140# Network: 2 inputs → 16 hidden → 16 hidden → 1 output
141nn = NeuralNetworkFromScratch(
142 layer_sizes=[2, 16, 16, 1],
143 learning_rate=0.1,
144 seed=42
145)
146
147print("Network Architecture:")
148for i in range(1, nn.num_layers):
149 W_shape = nn.parameters[f'W{i}'].shape
150 b_shape = nn.parameters[f'b{i}'].shape
151 print(f" Layer {i}: W{W_shape}, b{b_shape}")
152
153# ============================================================================
154# INITIAL FORWARD PASS
155# ============================================================================
156print("
1573. Testing Forward Propagation...")
158print("-" * 70)
159
160# Forward pass
161A_out, cache = nn.forward_propagation(X_train)
162
163print("Forward Propagation Cache:")
164for key in sorted(cache.keys()):
165 print(f" {key}: {cache[key].shape}")
166
167# Compute initial metrics
168initial_loss = nn.compute_loss(A_out, y_train)
169initial_accuracy = nn.accuracy(X_train, y_train)
170
171print(f"
172Initial Metrics:")
173print(f" Training Loss: {initial_loss:.4f}")
174print(f" Training Accuracy: {initial_accuracy * 100:.2f}%")
175print(f" Test Accuracy: {nn.accuracy(X_test, y_test) * 100:.2f}%")
176
177# ============================================================================
178# VISUALIZE NETWORK OUTPUT
179# ============================================================================
180print("
1814. Visualizing Network Behavior...")
182print("-" * 70)
183
184# Create meshgrid for decision boundary
185x_min, x_max = X_train[0, :].min() - 0.5, X_train[0, :].max() + 0.5
186y_min, y_max = X_train[1, :].min() - 0.5, X_train[1, :].max() + 0.5
187h = 0.02
188xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
189 np.arange(y_min, y_max, h))
190
191# Predict on meshgrid
192Z, _ = nn.forward_propagation(np.c_[xx.ravel(), yy.ravel()].T)
193Z = Z.reshape(xx.shape)
194
195# Create visualization
196fig, axes = plt.subplots(1, 3, figsize=(15, 4))
197
198# Plot 1: Dataset
199axes[0].scatter(X_train[0, y_train[0] == 0], X_train[1, y_train[0] == 0],
200 c='blue', marker='o', label='Class 0', alpha=0.6, s=30)
201axes[0].scatter(X_train[0, y_train[0] == 1], X_train[1, y_train[0] == 1],
202 c='red', marker='s', label='Class 1', alpha=0.6, s=30)
203axes[0].set_xlabel('Feature 1', fontweight='bold')
204axes[0].set_ylabel('Feature 2', fontweight='bold')
205axes[0].set_title('Training Dataset', fontweight='bold')
206axes[0].legend()
207axes[0].grid(True, alpha=0.3)
208
209# Plot 2: Initial Decision Boundary
210axes[1].contourf(xx, yy, Z, levels=20, cmap='RdBu', alpha=0.6)
211axes[1].scatter(X_train[0, y_train[0] == 0], X_train[1, y_train[0] == 0],
212 c='blue', marker='o', edgecolors='black', s=30)
213axes[1].scatter(X_train[0, y_train[0] == 1], X_train[1, y_train[0] == 1],
214 c='red', marker='s', edgecolors='black', s=30)
215axes[1].set_xlabel('Feature 1', fontweight='bold')
216axes[1].set_ylabel('Feature 2', fontweight='bold')
217axes[1].set_title('Initial Decision Boundary (Random Weights)', fontweight='bold')
218axes[1].grid(True, alpha=0.3)
219
220# Plot 3: Activation Distribution
221axes[2].hist(A_out.flatten(), bins=30, edgecolor='black', alpha=0.7)
222axes[2].axvline(0.5, color='red', linestyle='--', linewidth=2, label='Threshold')
223axes[2].set_xlabel('Predicted Probability', fontweight='bold')
224axes[2].set_ylabel('Frequency', fontweight='bold')
225axes[2].set_title('Output Distribution', fontweight='bold')
226axes[2].legend()
227axes[2].grid(True, alpha=0.3)
228
229plt.tight_layout()
230plt.show()
231
232# ============================================================================
233# LAYER-BY-LAYER ANALYSIS
234# ============================================================================
235print("
2365. Layer-by-Layer Analysis...")
237print("-" * 70)
238
239# Analyze each layer's output
240sample_input = X_train[:, :5] # First 5 samples
241print(f"Input shape: {sample_input.shape}")
242print(f"Input data (first 2 samples):")
243print(sample_input[:, :2])
244
245A_out_sample, cache_sample = nn.forward_propagation(sample_input)
246
247print(f"
248Layer-by-layer activations:")
249for i in range(nn.num_layers):
250 A = cache_sample[f'A{i}']
251 print(f" Layer {i} output: {A.shape}")
252 if i > 0:
253 active_neurons = np.sum(A > 0, axis=0)
254 print(f" Active neurons (sample 1): {active_neurons[0]}/{A.shape[0]}")
255
256# ============================================================================
257# SUMMARY
258# ============================================================================
259print("
260" + "=" * 70)
261print("NEURAL NETWORK SUMMARY")
262print("=" * 70)
263print(f"
264Architecture:")
265print(f" Input Layer: {nn.layer_sizes[0]} neurons")
266for i in range(1, len(nn.layer_sizes) - 1):
267 print(f" Hidden Layer {i}: {nn.layer_sizes[i]} neurons (ReLU)")
268print(f" Output Layer: {nn.layer_sizes[-1]} neuron (Sigmoid)")
269
270print(f"
271Parameters:")
272total_params = sum(
273 nn.parameters[f'W{i}'].size + nn.parameters[f'b{i}'].size
274 for i in range(1, nn.num_layers)
275)
276print(f" Total Parameters: {total_params:,}")
277
278print(f"
279Performance (Untrained):")
280print(f" Training Loss: {initial_loss:.4f}")
281print(f" Training Accuracy: {initial_accuracy * 100:.2f}%")
282print(f" Test Accuracy: {nn.accuracy(X_test, y_test) * 100:.2f}%")
283
284print("
285" + "=" * 70)
286print("Note: This network has random weights (no training yet)")
287print("Next lesson: Backpropagation to train the network!")
288print("=" * 70)This comprehensive implementation demonstrates building a complete neural network from scratch:
🏗️ Network Architecture:
Layer Structure:
Why This Architecture?
🔧 Key Implementation Details:
1. Weight Initialization (He Initialization):
1W = np.random.randn(n_out, n_in) * np.sqrt(2.0 / n_in)2. Forward Propagation Flow:
1Input (2,) → [W1, b1] → ReLU → Hidden1 (16,)
2 → [W2, b2] → ReLU → Hidden2 (16,)
3 → [W3, b3] → Sigmoid → Output (1,)3. Matrix Dimensions:
📊 Dataset (Make Moons):
Why Make Moons?
Data Shape:
🎯 Forward Propagation Process:
Hidden Layers:
Output Layer:
📈 Visualizations:
Plot 1 - Training Dataset:
Plot 2 - Decision Boundary:
Plot 3 - Output Distribution:
🔍 Layer Analysis:
Activation Patterns:
Parameter Count:
🎓 Key Concepts Demonstrated:
1. Object-Oriented Design:
2. Vectorization:
3. Numerical Stability:
4. Caching:
💡 Performance Analysis:
Before Training (Random Weights):
Expected After Training:
🚀 Next Steps:
This implementation provides the foundation for:
Best Practices Shown:
Excellent work building neural networks from scratch! You now understand perceptrons, activations, forward propagation, and loss functions. In our next lesson, "Backpropagation and Optimization," you'll learn to train networks by computing gradients through backpropagation, implement gradient descent variants (SGD, Adam, RMSprop), and optimize learning with techniques like learning rate scheduling. Get ready to make your networks learn!
Your progress is saved in this browser, on this device only.