Master sequential data processing with RNNs. Learn hidden state, LSTM gates, GRU architecture, and sequence-to-sequence models for text generation, time series forecasting, and language modeling.
So far, we've focused on spatial data—images where the order of pixels within local regions matters. But what about temporal data where the order of elements through time is crucial?
Consider these problems:
Traditional neural networks have a fundamental limitation: they don't have memory. Each input is processed independently, with no knowledge of what came before or after. This is fine for image classification, but fails completely for sequences.
Enter Recurrent Neural Networks (RNNs):
RNNs solve this by maintaining a hidden state—a memory that persists across time steps. Each time the RNN processes an input, it:
This simple mechanism enables RNNs to:
However, vanilla RNNs face the vanishing gradient problem: gradients shrink exponentially during backpropagation through time, making it nearly impossible to learn long-term dependencies.
LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Units) solved this problem with clever gating mechanisms that allow gradients to flow unchanged across hundreds of time steps.
In this lesson, you'll understand how RNNs process sequences, why vanilla RNNs fail on long sequences, and how LSTM/GRU gates enable learning of long-term dependencies. You'll implement character-level text generation, time series forecasting, and sequence-to-sequence translation.
Ready to give neural networks memory? Let's dive into the temporal dimension! 🕐
Recurrent Neural Networks process sequences by maintaining a hidden state that gets updated at each time step.
Problem 1: Fixed Input Size
1# Traditional neural network requires fixed-size input
2model = Sequential([
3 Dense(128, input_shape=(10,)), # Must be exactly 10 features
4 Dense(64),
5 Dense(1)
6])
7
8# ❌ Cannot handle:
9# - Sentences of different lengths
10# - Time series of varying duration
11# - Video clips with different frame countsProblem 2: No Temporal Memory
1# Traditional network processes each input independently
2sentence = ["The", "cat", "sat"]
3
4for word in sentence:
5 embedding = get_word_embedding(word)
6 output = model(embedding)
7 # ❌ No memory of previous words!
8 # Model doesn't know "The" came before "cat"Core Idea: Maintain a hidden state h_t that captures context from previous time steps.
1At time step t:
2 h_t = f(x_t, h_{t-1})
3
4Where:
5 x_t = current input
6 h_{t-1} = previous hidden state (memory)
7 h_t = new hidden state
8 f = recurrent function1import numpy as np
2
3def rnn_step(x_t, h_prev, W_xh, W_hh, b_h):
4 """
5 Single RNN time step
6
7 Args:
8 x_t: Input at time t (shape: input_size)
9 h_prev: Previous hidden state (shape: hidden_size)
10 W_xh: Input-to-hidden weights
11 W_hh: Hidden-to-hidden weights (recurrence!)
12 b_h: Hidden bias
13
14 Returns:
15 h_t: New hidden state
16 """
17 h_t = np.tanh(
18 np.dot(W_xh, x_t) + # Input contribution
19 np.dot(W_hh, h_prev) + # Previous state contribution
20 b_h # Bias
21 )
22 return h_t
23
24# Example: Process sequence "hello"
25vocab = {c: i for i, c in enumerate("helo ")}
26sequence = "hello"
27
28# Initialize
29hidden_size = 128
30input_size = len(vocab)
31
32# Random weights (normally learned during training)
33W_xh = np.random.randn(hidden_size, input_size) * 0.01
34W_hh = np.random.randn(hidden_size, hidden_size) * 0.01
35b_h = np.zeros(hidden_size)
36
37# Initial hidden state (zeros)
38h = np.zeros(hidden_size)
39
40print("Processing sequence 'hello' through RNN:")
41print("=" * 60)
42
43for t, char in enumerate(sequence):
44 # One-hot encode character
45 x_t = np.zeros(input_size)
46 x_t[vocab[char]] = 1
47
48 # RNN step
49 h = rnn_step(x_t, h, W_xh, W_hh, b_h)
50
51 print(f"t={t}, char='{char}', hidden_state_norm={np.linalg.norm(h):.4f}")
52
53print("=" * 60)
54print("Final hidden state captures context of entire sequence!")Output:
1Processing sequence 'hello' through RNN:
2============================================================
3t=0, char='h', hidden_state_norm=0.3214
4t=1, char='e', hidden_state_norm=0.4567
5t=2, char='l', hidden_state_norm=0.5123
6t=3, char='l', hidden_state_norm=0.5456
7t=4, char='o', hidden_state_norm=0.5789
8============================================================
9Final hidden state captures context of entire sequence!1def rnn_forward(X, h_0, W_xh, W_hh, W_hy, b_h, b_y):
2 """
3 Full RNN forward pass through entire sequence
4
5 Args:
6 X: Input sequence (time_steps, input_size)
7 h_0: Initial hidden state
8 W_xh, W_hh: RNN weights
9 W_hy: Hidden-to-output weights
10 b_h, b_y: Biases
11
12 Returns:
13 outputs: Output at each time step
14 hidden_states: Hidden state at each time step
15 """
16 time_steps = X.shape[0]
17 hidden_size = h_0.shape[0]
18 output_size = W_hy.shape[0]
19
20 # Store all hidden states and outputs
21 hidden_states = np.zeros((time_steps, hidden_size))
22 outputs = np.zeros((time_steps, output_size))
23
24 h = h_0
25
26 # Process sequence step by step
27 for t in range(time_steps):
28 # RNN step
29 h = np.tanh(
30 np.dot(W_xh, X[t]) +
31 np.dot(W_hh, h) +
32 b_h
33 )
34
35 # Output (e.g., next character prediction)
36 y = np.dot(W_hy, h) + b_y
37
38 hidden_states[t] = h
39 outputs[t] = y
40
41 return outputs, hidden_states
42
43# Visualize
44print("\nRNN Architecture (Unrolled):")
45print("""
46 x_0 x_1 x_2 x_3 x_4
47 | | | | |
48 ↓ ↓ ↓ ↓ ↓
49 [RNN] → [RNN] → [RNN] → [RNN] → [RNN]
50 | | | | |
51 ↓ ↓ ↓ ↓ ↓
52 y_0 y_1 y_2 y_3 y_4
53
54Key: → represents hidden state flowing through time
55 ↓ represents outputs at each time step
56""")1. Many-to-Many (Sequence-to-Sequence)
1Example: Video frame captioning
2Input: [frame_1, frame_2, ..., frame_n]
3Output: [word_1, word_2, ..., word_m]2. Many-to-One (Sequence-to-Vector)
1Example: Sentiment analysis
2Input: ["This", "movie", "is", "great"]
3Output: [positive_sentiment]3. One-to-Many (Vector-to-Sequence)
1Example: Image captioning
2Input: [image_features]
3Output: ["A", "cat", "on", "a", "mat"]Training RNNs requires computing gradients across time:
1def bptt_simplified(outputs, targets, hidden_states, W_hy, W_hh):
2 """
3 Simplified BPTT to show gradient flow
4
5 Key insight: Gradients flow backward through time,
6 multiplied by W_hh at each step
7 """
8 time_steps = len(outputs)
9 dh_next = np.zeros_like(hidden_states[0])
10
11 gradients = []
12
13 for t in reversed(range(time_steps)):
14 # Output gradient
15 dy = outputs[t] - targets[t]
16
17 # Hidden state gradient (from output + future)
18 dh = np.dot(W_hy.T, dy) + dh_next
19
20 # Gradient flows through time
21 dh_next = np.dot(W_hh.T, dh) * (1 - hidden_states[t]**2) # tanh derivative
22
23 gradients.append(np.linalg.norm(dh))
24
25 return gradients[::-1]
26
27# Show gradient magnitude through time
28print("\nGradient magnitudes during BPTT:")
29print("(Notice how gradients can vanish/explode)")1# Simulate gradient flow through many time steps
2def gradient_flow_simulation(time_steps=50):
3 """
4 Show how gradients vanish in vanilla RNN
5 """
6 gradient = 1.0
7 W_hh_value = 0.9 # Typical weight magnitude
8
9 gradients = [gradient]
10
11 for t in range(time_steps):
12 # Gradient *= W_hh at each time step
13 gradient *= W_hh_value
14 gradients.append(gradient)
15
16 return gradients
17
18gradients = gradient_flow_simulation(50)
19
20print(f"\nGradient after 50 time steps: {gradients[-1]:.10f}")
21print("This is why vanilla RNNs struggle with long sequences!")
22print("Solution: LSTM and GRU architectures →")Advantages:
✓ Handle variable-length sequences
✓ Parameter sharing across time steps
✓ Maintain temporal dependencies
✓ Model sequential patterns
Challenges:
✗ Vanishing/exploding gradients
✗ Difficulty learning long-term dependencies (>20 steps)
✗ Sequential processing (slow, can't parallelize)
✗ Struggle to capture very long-range dependencies
Solution: LSTM and GRU add gating mechanisms to control information flow!
LSTM solves the vanishing gradient problem through carefully designed gates that control information flow.
Key Idea: Separate long-term memory (cell state C_t) from short-term memory (hidden state h_t), and use gates to control what gets remembered, forgotten, and output.
1LSTM has TWO states:
21. Cell State (C_t): Long-term memory highway
32. Hidden State (h_t): Short-term working memory1. Forget Gate: What to forget from cell state
2. Input Gate: What new information to store
3. Output Gate: What to output from cell state
1import numpy as np
2
3def lstm_step(x_t, h_prev, C_prev, W_f, W_i, W_C, W_o, b_f, b_i, b_C, b_o):
4 """
5 Single LSTM time step
6
7 Args:
8 x_t: Current input
9 h_prev: Previous hidden state
10 C_prev: Previous cell state
11 W_f, W_i, W_C, W_o: Weight matrices for gates
12 b_f, b_i, b_C, b_o: Bias vectors for gates
13
14 Returns:
15 h_t: New hidden state
16 C_t: New cell state
17 """
18 # Concatenate input and previous hidden state
19 concat = np.concatenate([h_prev, x_t])
20
21 # 1. FORGET GATE: Decide what to forget from C_prev
22 # Sigmoid → [0, 1] where 0 = forget, 1 = keep
23 f_t = sigmoid(np.dot(W_f, concat) + b_f)
24
25 # 2. INPUT GATE: Decide what new information to store
26 # i_t: Which values to update (sigmoid)
27 # C_tilde: Candidate values to add (tanh)
28 i_t = sigmoid(np.dot(W_i, concat) + b_i)
29 C_tilde = np.tanh(np.dot(W_C, concat) + b_C)
30
31 # 3. UPDATE CELL STATE: Forget old info + add new info
32 C_t = f_t * C_prev + i_t * C_tilde
33
34 # 4. OUTPUT GATE: Decide what to output
35 o_t = sigmoid(np.dot(W_o, concat) + b_o)
36 h_t = o_t * np.tanh(C_t)
37
38 return h_t, C_t
39
40def sigmoid(x):
41 return 1 / (1 + np.exp(-x))
42
43# Demonstrate LSTM step by step
44print("LSTM Step-by-Step Computation:")
45print("=" * 70)
46
47# Example dimensions
48hidden_size = 4
49input_size = 3
50
51# Initialize states
52h_prev = np.random.randn(hidden_size)
53C_prev = np.random.randn(hidden_size)
54x_t = np.random.randn(input_size)
55
56# Initialize weights (simplified)
57concat_size = hidden_size + input_size
58W_f = np.random.randn(hidden_size, concat_size) * 0.1
59W_i = np.random.randn(hidden_size, concat_size) * 0.1
60W_C = np.random.randn(hidden_size, concat_size) * 0.1
61W_o = np.random.randn(hidden_size, concat_size) * 0.1
62
63b_f = np.zeros(hidden_size)
64b_i = np.zeros(hidden_size)
65b_C = np.zeros(hidden_size)
66b_o = np.zeros(hidden_size)
67
68# Compute LSTM step
69concat = np.concatenate([h_prev, x_t])
70
71f_t = sigmoid(np.dot(W_f, concat) + b_f)
72i_t = sigmoid(np.dot(W_i, concat) + b_i)
73C_tilde = np.tanh(np.dot(W_C, concat) + b_C)
74o_t = sigmoid(np.dot(W_o, concat) + b_o)
75
76C_t = f_t * C_prev + i_t * C_tilde
77h_t = o_t * np.tanh(C_t)
78
79print(f"Forget gate (f_t): {f_t}")
80print(f" → Controls what to forget from previous cell state")
81print(f" → Values near 0 = forget, near 1 = keep")
82print()
83print(f"Input gate (i_t): {i_t}")
84print(f" → Controls what new information to add")
85print()
86print(f"Candidate (C_tilde): {C_tilde}")
87print(f" → New candidate values to potentially add to cell state")
88print()
89print(f"Output gate (o_t): {o_t}")
90print(f" → Controls what to output from cell state")
91print()
92print(f"New cell state (C_t): {C_t}")
93print(f" → Long-term memory")
94print()
95print(f"New hidden state (h_t): {h_t}")
96print(f" → Short-term output")
97print("=" * 70)1import matplotlib.pyplot as plt
2
3def visualize_lstm_gates():
4 """
5 Visualize how LSTM gates work
6 """
7 time_steps = 20
8
9 # Simulate gate activations over time
10 np.random.seed(42)
11 forget_gate = 0.5 + 0.3 * np.sin(np.linspace(0, 4*np.pi, time_steps)) + np.random.randn(time_steps) * 0.1
12 input_gate = 0.6 + 0.2 * np.cos(np.linspace(0, 4*np.pi, time_steps)) + np.random.randn(time_steps) * 0.1
13 output_gate = 0.7 + 0.2 * np.sin(np.linspace(0, 2*np.pi, time_steps)) + np.random.randn(time_steps) * 0.1
14
15 # Clip to [0, 1]
16 forget_gate = np.clip(forget_gate, 0, 1)
17 input_gate = np.clip(input_gate, 0, 1)
18 output_gate = np.clip(output_gate, 0, 1)
19
20 # Plot
21 fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
22
23 # Gate activations
24 time_axis = np.arange(time_steps)
25 ax1.plot(time_axis, forget_gate, 'r-o', linewidth=2, markersize=6, label='Forget Gate')
26 ax1.plot(time_axis, input_gate, 'g-o', linewidth=2, markersize=6, label='Input Gate')
27 ax1.plot(time_axis, output_gate, 'b-o', linewidth=2, markersize=6, label='Output Gate')
28 ax1.axhline(y=0.5, color='gray', linestyle='--', alpha=0.5)
29 ax1.set_xlabel('Time Step', fontweight='bold', fontsize=12)
30 ax1.set_ylabel('Gate Activation', fontweight='bold', fontsize=12)
31 ax1.set_title('LSTM Gate Activations Over Time', fontweight='bold', fontsize=14)
32 ax1.legend(fontsize=11, loc='upper right')
33 ax1.grid(True, alpha=0.3)
34 ax1.set_ylim(-0.1, 1.1)
35
36 # Cell state evolution
37 cell_state = np.zeros(time_steps)
38 cell_state[0] = 0.5
39
40 for t in range(1, time_steps):
41 # C_t = f_t * C_{t-1} + i_t * C_tilde
42 C_tilde = np.random.randn() * 0.5 # Candidate
43 cell_state[t] = forget_gate[t] * cell_state[t-1] + input_gate[t] * C_tilde
44
45 ax2.plot(time_axis, cell_state, 'purple', linewidth=3, marker='s', markersize=7)
46 ax2.fill_between(time_axis, 0, cell_state, alpha=0.3, color='purple')
47 ax2.axhline(y=0, color='black', linestyle='-', linewidth=1)
48 ax2.set_xlabel('Time Step', fontweight='bold', fontsize=12)
49 ax2.set_ylabel('Cell State Value', fontweight='bold', fontsize=12)
50 ax2.set_title('LSTM Cell State Evolution (Long-Term Memory)', fontweight='bold', fontsize=14)
51 ax2.grid(True, alpha=0.3)
52
53 plt.tight_layout()
54 plt.show()
55
56visualize_lstm_gates()
57
58print("Gate Interpretation:")
59print(" • Forget gate ~0: Erasing previous memory")
60print(" • Forget gate ~1: Retaining previous memory")
61print(" • Input gate ~1: Strongly writing new information")
62print(" • Output gate ~1: Exposing cell state to output")The Cell State Highway:
1Traditional RNN gradient flow:
2gradient_{t-n} = gradient_t × W^n
3→ Exponential decay if |W| < 1
4
5LSTM gradient flow:
6gradient flows through cell state with only element-wise multiplication
7→ No matrix multiplication = no vanishing!Mathematical Insight:
1# Gradient flow in vanilla RNN
2def rnn_gradient_flow(n_steps):
3 W = 0.9 # Weight matrix value
4 gradient = 1.0
5 for _ in range(n_steps):
6 gradient *= W
7 return gradient
8
9# Gradient flow in LSTM
10def lstm_gradient_flow(n_steps):
11 gradient = 1.0
12 for _ in range(n_steps):
13 # Forget gate allows gradient to flow unchanged
14 f_t = 0.95 # High forget gate = keep gradient
15 gradient *= f_t
16 return gradient
17
18steps = 50
19print("\nGradient Magnitudes After 50 Time Steps:")
20print("=" * 60)
21print(f"Vanilla RNN: {rnn_gradient_flow(steps):.10f}")
22print(f"LSTM: {lstm_gradient_flow(steps):.10f}")
23print("=" * 60)
24print("\nLSTM gradients remain strong over long sequences!")1import tensorflow as tf
2
3# Simple LSTM model for sequence classification
4model = tf.keras.Sequential([
5 # LSTM layer
6 tf.keras.layers.LSTM(
7 units=128, # Hidden state size
8 return_sequences=True, # Return full sequence (not just last output)
9 dropout=0.2, # Dropout for regularization
10 input_shape=(None, input_dim) # (time_steps, features)
11 ),
12
13 # Second LSTM layer
14 tf.keras.layers.LSTM(
15 units=64,
16 return_sequences=False # Return only last output
17 ),
18
19 # Classification head
20 tf.keras.layers.Dense(32, activation='relu'),
21 tf.keras.layers.Dropout(0.3),
22 tf.keras.layers.Dense(num_classes, activation='softmax')
23])
24
25# Compile
26model.compile(
27 optimizer='adam',
28 loss='categorical_crossentropy',
29 metrics=['accuracy']
30)
31
32print("LSTM Model Summary:")
33model.summary()Process sequences in both directions for richer context:
1# Bidirectional LSTM
2model = tf.keras.Sequential([
3 tf.keras.layers.Bidirectional(
4 tf.keras.layers.LSTM(128, return_sequences=True)
5 ),
6 # Output has 256 features (128 forward + 128 backward)
7 tf.keras.layers.LSTM(64),
8 tf.keras.layers.Dense(num_classes, activation='softmax')
9])
10
11# Use case: Sentiment analysis
12# "The movie was not good" → Process forward and backward
13# Forward: [The] [movie] [was] [not] [good]
14# Backward: [good] [not] [was] [movie] [The]
15# "not good" understood better with bidirectional context!Why LSTM Works:
When to Use LSTM:
✓ Long sequences (>20 time steps)
✓ Need to remember long-term dependencies
✓ Text generation, language modeling
✓ Time series with seasonal patterns
✓ Any sequential data with important long-range correlations
LSTM Limitations:
✗ Slower than vanilla RNN (4× more parameters per gate)
✗ Still sequential (can't parallelize like Transformers)
✗ May overfit on small datasets
✗ Requires more training data than simpler models
Next: GRU as a simpler, faster alternative!
GRU simplifies LSTM by merging the cell state and hidden state into a single state, achieving comparable performance with fewer parameters.
Key Difference from LSTM:
GRU Performance:
1. Reset Gate (r_t): Controls how much past information to forget
2. Update Gate (z_t): Controls how much to update vs keep previous state
1def gru_step(x_t, h_prev, W_r, W_z, W_h, b_r, b_z, b_h):
2 """
3 Single GRU time step
4
5 Args:
6 x_t: Current input
7 h_prev: Previous hidden state
8 W_r, W_z, W_h: Weight matrices
9 b_r, b_z, b_h: Bias vectors
10
11 Returns:
12 h_t: New hidden state (single state!)
13 """
14 # Concatenate input and previous hidden
15 concat = np.concatenate([h_prev, x_t])
16
17 # 1. RESET GATE: How much past info to use for candidate
18 r_t = sigmoid(np.dot(W_r, concat) + b_r)
19
20 # 2. UPDATE GATE: How much to update (vs keep previous)
21 z_t = sigmoid(np.dot(W_z, concat) + b_z)
22
23 # 3. CANDIDATE: New candidate state (uses reset gate)
24 concat_reset = np.concatenate([r_t * h_prev, x_t])
25 h_tilde = np.tanh(np.dot(W_h, concat_reset) + b_h)
26
27 # 4. FINAL STATE: Interpolate between previous and candidate
28 # z_t = 1 → keep previous (h_prev)
29 # z_t = 0 → use candidate (h_tilde)
30 h_t = (1 - z_t) * h_prev + z_t * h_tilde
31
32 return h_t
33
34def sigmoid(x):
35 return 1 / (1 + np.exp(-x))
36
37# Example GRU computation
38print("GRU Step-by-Step:")
39print("=" * 70)
40
41hidden_size = 4
42input_size = 3
43
44h_prev = np.array([0.5, -0.3, 0.8, 0.2])
45x_t = np.array([1.0, 0.5, -0.5])
46
47# Initialize weights
48concat_size = hidden_size + input_size
49W_r = np.random.randn(hidden_size, concat_size) * 0.1
50W_z = np.random.randn(hidden_size, concat_size) * 0.1
51W_h = np.random.randn(hidden_size, concat_size) * 0.1
52b_r = np.zeros(hidden_size)
53b_z = np.zeros(hidden_size)
54b_h = np.zeros(hidden_size)
55
56# Compute
57concat = np.concatenate([h_prev, x_t])
58r_t = sigmoid(np.dot(W_r, concat) + b_r)
59z_t = sigmoid(np.dot(W_z, concat) + b_z)
60
61concat_reset = np.concatenate([r_t * h_prev, x_t])
62h_tilde = np.tanh(np.dot(W_h, concat_reset) + b_h)
63
64h_t = (1 - z_t) * h_prev + z_t * h_tilde
65
66print(f"Previous state (h_prev): {h_prev}")
67print()
68print(f"Reset gate (r_t): {r_t}")
69print(f" → Controls how much past to use in candidate")
70print(f" → Low values = forget past, focus on current input")
71print()
72print(f"Update gate (z_t): {z_t}")
73print(f" → Controls keep vs update")
74print(f" → High values = update with new candidate")
75print()
76print(f"Candidate (h_tilde): {h_tilde}")
77print(f" → New proposed state")
78print()
79print(f"Final state (h_t): {h_t}")
80print(f" → Interpolation: (1-z)*h_prev + z*h_tilde")
81print("=" * 70)1# Comparison table
2comparison = """
3┌─────────────────────┬────────────────────────┬────────────────────────┐
4│ Feature │ LSTM │ GRU │
5├─────────────────────┼────────────────────────┼────────────────────────┤
6│ States │ 2 (cell + hidden) │ 1 (hidden only) │
7│ Gates │ 3 (forget/input/output)│ 2 (reset/update) │
8│ Parameters │ 4n² + 4nm │ 3n² + 3nm │
9│ Training Speed │ Baseline (1×) │ 1.2-1.3× faster │
10│ Memory Usage │ Higher │ 25-33% lower │
11│ Accuracy │ Baseline (100%) │ 92-98% of LSTM │
12│ Ease of Tuning │ More hyperparameters │ Simpler, fewer params │
13│ Best For │ Very long sequences │ Most sequence tasks │
14│ │ Complex dependencies │ Faster prototyping │
15└─────────────────────┴────────────────────────┴────────────────────────┘
16
17Where:
18 n = hidden size
19 m = input size
20"""
21print(comparison)1import tensorflow as tf
2
3# GRU model (simpler than LSTM)
4model = tf.keras.Sequential([
5 # GRU layer
6 tf.keras.layers.GRU(
7 units=128,
8 return_sequences=True,
9 dropout=0.2,
10 input_shape=(None, input_dim)
11 ),
12
13 # Second GRU layer
14 tf.keras.layers.GRU(
15 units=64,
16 return_sequences=False
17 ),
18
19 # Classification
20 tf.keras.layers.Dense(32, activation='relu'),
21 tf.keras.layers.Dense(num_classes, activation='softmax')
22])
23
24model.compile(
25 optimizer='adam',
26 loss='categorical_crossentropy',
27 metrics=['accuracy']
28)
29
30print("GRU Model:")
31print(f"Parameters: {model.count_params():,}")
32print(f"(LSTM would have ~33% more parameters)")1import matplotlib.pyplot as plt
2
3def visualize_gru_behavior():
4 """
5 Show how GRU gates control information flow
6 """
7 time_steps = 20
8
9 # Simulate gate behavior
10 np.random.seed(42)
11 reset_gate = 0.3 + 0.4 * np.sin(np.linspace(0, 3*np.pi, time_steps)) + np.random.randn(time_steps) * 0.1
12 update_gate = 0.5 + 0.3 * np.cos(np.linspace(0, 3*np.pi, time_steps)) + np.random.randn(time_steps) * 0.1
13
14 reset_gate = np.clip(reset_gate, 0, 1)
15 update_gate = np.clip(update_gate, 0, 1)
16
17 fig, axes = plt.subplots(3, 1, figsize=(14, 12))
18 time_axis = np.arange(time_steps)
19
20 # Gates
21 axes[0].plot(time_axis, reset_gate, 'b-o', linewidth=2, markersize=6, label='Reset Gate (r_t)')
22 axes[0].plot(time_axis, update_gate, 'g-o', linewidth=2, markersize=6, label='Update Gate (z_t)')
23 axes[0].axhline(y=0.5, color='gray', linestyle='--', alpha=0.5)
24 axes[0].set_ylabel('Gate Value', fontweight='bold', fontsize=12)
25 axes[0].set_title('GRU Gate Activations', fontweight='bold', fontsize=14)
26 axes[0].legend(fontsize=11)
27 axes[0].grid(True, alpha=0.3)
28 axes[0].set_ylim(-0.1, 1.1)
29
30 # Hidden state evolution
31 h = np.zeros(time_steps)
32 h[0] = 0.5
33
34 for t in range(1, time_steps):
35 # Simplified GRU update
36 h_tilde = np.random.randn() * 0.5 # Candidate
37 h[t] = (1 - update_gate[t]) * h[t-1] + update_gate[t] * h_tilde
38
39 axes[1].plot(time_axis, h, 'purple', linewidth=3, marker='s', markersize=7)
40 axes[1].fill_between(time_axis, 0, h, alpha=0.3, color='purple')
41 axes[1].axhline(y=0, color='black', linestyle='-', linewidth=1)
42 axes[1].set_ylabel('Hidden State', fontweight='bold', fontsize=12)
43 axes[1].set_title('GRU Hidden State Evolution', fontweight='bold', fontsize=14)
44 axes[1].grid(True, alpha=0.3)
45
46 # Update vs Keep ratio
47 keep_ratio = 1 - update_gate
48 axes[2].bar(time_axis, update_gate, alpha=0.7, label='Update (new)', color='green', edgecolor='black')
49 axes[2].bar(time_axis, keep_ratio, bottom=update_gate, alpha=0.7, label='Keep (previous)', color='orange', edgecolor='black')
50 axes[2].set_xlabel('Time Step', fontweight='bold', fontsize=12)
51 axes[2].set_ylabel('Proportion', fontweight='bold', fontsize=12)
52 axes[2].set_title('Update vs Keep Previous State', fontweight='bold', fontsize=14)
53 axes[2].legend(fontsize=11)
54 axes[2].set_ylim(0, 1)
55
56 plt.tight_layout()
57 plt.show()
58
59visualize_gru_behavior()
60
61print("\nGRU Gate Interpretation:")
62print(" • Reset gate ~0: Ignore past, focus on current input")
63print(" • Reset gate ~1: Use past information in candidate")
64print(" • Update gate ~0: Keep previous state (ignore candidate)")
65print(" • Update gate ~1: Update with new candidate (forget previous)")Use GRU when:
✓ Faster training is priority
✓ Limited computational resources
✓ Prototyping/experimentation phase
✓ Moderate sequence lengths (<100 steps)
✓ Dataset is small-to-medium
Use LSTM when:
✓ Very long sequences (>100 steps)
✓ Complex long-term dependencies
✓ Maximum accuracy is critical
✓ Large dataset available
✓ Proven LSTM architecture exists for your task
Real-World Usage (2025):
1# Deep GRU network
2model = tf.keras.Sequential([
3 tf.keras.layers.GRU(256, return_sequences=True, input_shape=(None, input_dim)),
4 tf.keras.layers.Dropout(0.3),
5
6 tf.keras.layers.GRU(256, return_sequences=True),
7 tf.keras.layers.Dropout(0.3),
8
9 tf.keras.layers.GRU(128, return_sequences=True),
10 tf.keras.layers.Dropout(0.3),
11
12 tf.keras.layers.GRU(64),
13
14 tf.keras.layers.Dense(num_classes, activation='softmax')
15])
16
17# Deeper networks can learn more complex patterns
18# But require more data and careful regularizationGRU is the "good enough" solution:
Unless you have specific reasons to use LSTM, start with GRU!
Sequence-to-sequence (seq2seq) models revolutionized machine translation, text summarization, and conversational AI by learning to map variable-length input sequences to variable-length output sequences.
Core Idea: Encoder-Decoder architecture
1Input: "Hello world" (English)
2 ↓
3 Encoder (LSTM/GRU)
4 ↓
5 Context Vector (thought)
6 ↓
7 Decoder (LSTM/GRU)
8 ↓
9Output: "Bonjour le monde" (French)1import tensorflow as tf
2
3class Seq2SeqModel:
4 """
5 Basic sequence-to-sequence model
6 Encoder-decoder architecture with LSTM
7 """
8
9 def __init__(self, vocab_size_in, vocab_size_out, embedding_dim=256, hidden_size=512):
10 self.vocab_size_in = vocab_size_in
11 self.vocab_size_out = vocab_size_out
12 self.embedding_dim = embedding_dim
13 self.hidden_size = hidden_size
14
15 # Encoder
16 self.encoder_embedding = tf.keras.layers.Embedding(vocab_size_in, embedding_dim)
17 self.encoder_lstm = tf.keras.layers.LSTM(hidden_size, return_state=True)
18
19 # Decoder
20 self.decoder_embedding = tf.keras.layers.Embedding(vocab_size_out, embedding_dim)
21 self.decoder_lstm = tf.keras.layers.LSTM(hidden_size, return_sequences=True, return_state=True)
22 self.decoder_dense = tf.keras.layers.Dense(vocab_size_out, activation='softmax')
23
24 def encode(self, input_sequence):
25 """
26 Encode input sequence into context vector
27
28 Args:
29 input_sequence: (batch_size, input_length)
30
31 Returns:
32 encoder_state: (h, c) for LSTM
33 """
34 # Embed input
35 embedded = self.encoder_embedding(input_sequence)
36
37 # Encode (only keep final state)
38 _, h, c = self.encoder_lstm(embedded)
39
40 return h, c
41
42 def decode(self, target_sequence, initial_state):
43 """
44 Decode target sequence given initial state
45
46 Args:
47 target_sequence: (batch_size, target_length)
48 initial_state: Encoder final state (h, c)
49
50 Returns:
51 predictions: (batch_size, target_length, vocab_size)
52 """
53 # Embed target
54 embedded = self.decoder_embedding(target_sequence)
55
56 # Decode
57 outputs, _, _ = self.decoder_lstm(embedded, initial_state=initial_state)
58
59 # Project to vocabulary
60 predictions = self.decoder_dense(outputs)
61
62 return predictions
63
64 def call(self, input_sequence, target_sequence):
65 """
66 Full forward pass: encode input, decode target
67 """
68 # Encode
69 encoder_state = self.encode(input_sequence)
70
71 # Decode
72 predictions = self.decode(target_sequence, encoder_state)
73
74 return predictions
75
76# Example usage
77vocab_size_in = 10000 # English vocabulary
78vocab_size_out = 8000 # French vocabulary
79
80model = Seq2SeqModel(vocab_size_in, vocab_size_out)
81
82print("Seq2Seq Model Architecture:")
83print(f" Encoder: {vocab_size_in} tokens → Embedding(256) → LSTM(512)")
84print(f" Context: Fixed 512-dimensional vector")
85print(f" Decoder: LSTM(512) → Dense({vocab_size_out})")Problem: During training, decoder makes mistakes early, which compound over time.
Solution: Teacher forcing—feed ground truth at each step (not model's previous prediction).
1def train_step(model, input_seq, target_seq):
2 """
3 Single training step with teacher forcing
4
5 Args:
6 input_seq: Source language sentence
7 target_seq: Target language sentence (ground truth)
8 """
9 with tf.GradientTape() as tape:
10 # Encode input
11 encoder_state = model.encode(input_seq)
12
13 # Decoder input: shift target right by one (<START> token)
14 decoder_input = target_seq[:, :-1] # All but last
15 decoder_target = target_seq[:, 1:] # All but first
16
17 # Decode (teacher forcing: use ground truth as input)
18 predictions = model.decode(decoder_input, encoder_state)
19
20 # Compute loss
21 loss = tf.keras.losses.sparse_categorical_crossentropy(
22 decoder_target, predictions
23 )
24
25 # Update weights
26 gradients = tape.gradient(loss, model.trainable_variables)
27 optimizer.apply_gradients(zip(gradients, model.trainable_variables))
28
29 return loss
30
31print("Teacher Forcing Example:")
32print("Input: 'Hello world'")
33print("Target: '<START> Bonjour le monde <END>'")
34print()
35print("Decoder steps:")
36print(" Step 1: Input='<START>' → Predict='Bonjour' (teacher forces 'Bonjour')")
37print(" Step 2: Input='Bonjour' → Predict='le' (teacher forces 'le')")
38print(" Step 3: Input='le' → Predict='monde' (teacher forces 'monde')")
39print(" Step 4: Input='monde' → Predict='<END>'")At test time, we don't have ground truth. Generate output autoregressively:
1def greedy_decode(model, input_sequence, max_length=50):
2 """
3 Generate output sequence greedily (no teacher forcing)
4
5 Args:
6 input_sequence: Source sentence
7 max_length: Maximum output length
8
9 Returns:
10 output_sequence: Generated translation
11 """
12 # Encode input
13 encoder_state = model.encode(input_sequence)
14
15 # Start with <START> token
16 decoder_input = tf.constant([[START_TOKEN]])
17
18 output_sequence = []
19
20 for t in range(max_length):
21 # Decode one step
22 predictions = model.decode(decoder_input, encoder_state)
23
24 # Get most likely token (greedy)
25 predicted_token = tf.argmax(predictions[0, -1, :]).numpy()
26
27 # Stop if <END> token
28 if predicted_token == END_TOKEN:
29 break
30
31 # Append to output
32 output_sequence.append(predicted_token)
33
34 # Use prediction as next input (autoregressive)
35 decoder_input = tf.constant([[predicted_token]])
36
37 return output_sequence
38
39# Example
40input_text = "Hello world"
41output_tokens = greedy_decode(model, input_text)
42output_text = decode_tokens(output_tokens) # "Bonjour le monde"Greedy decoding is myopic. Beam search explores multiple hypotheses:
1def beam_search_decode(model, input_sequence, beam_width=5, max_length=50):
2 """
3 Beam search for better translation quality
4
5 Keeps top-k hypotheses at each step
6 """
7 # Encode
8 encoder_state = model.encode(input_sequence)
9
10 # Initialize beam: [(sequence, score)]
11 beams = [([START_TOKEN], 0.0)]
12
13 for t in range(max_length):
14 candidates = []
15
16 for sequence, score in beams:
17 # If already ended, keep as is
18 if sequence[-1] == END_TOKEN:
19 candidates.append((sequence, score))
20 continue
21
22 # Decode one step
23 decoder_input = tf.constant([sequence])
24 predictions = model.decode(decoder_input, encoder_state)
25
26 # Get top-k tokens
27 top_k_probs, top_k_tokens = tf.nn.top_k(
28 predictions[0, -1, :], k=beam_width
29 )
30
31 # Extend beam
32 for prob, token in zip(top_k_probs, top_k_tokens):
33 new_sequence = sequence + [token.numpy()]
34 new_score = score + tf.math.log(prob).numpy()
35 candidates.append((new_sequence, new_score))
36
37 # Keep top-k hypotheses
38 beams = sorted(candidates, key=lambda x: x[1], reverse=True)[:beam_width]
39
40 # Return best sequence
41 return beams[0][0]
42
43print("\nBeam Search (beam_width=3):")
44print("Step 1:")
45print(" Hypothesis 1: '<START> Bonjour' (score: -0.5)")
46print(" Hypothesis 2: '<START> Salut' (score: -0.8)")
47print(" Hypothesis 3: '<START> Hello' (score: -1.2)")
48print()
49print("Step 2 (expand each hypothesis):")
50print(" Hypothesis 1a: 'Bonjour le' (score: -0.7)")
51print(" Hypothesis 1b: 'Bonjour tout' (score: -1.1)")
52print(" Hypothesis 2a: 'Salut le' (score: -1.3)")
53print(" ... (keep top-3)")Basic seq2seq has a bottleneck: entire input must fit in fixed-size context vector.
Attention allows decoder to "look back" at encoder states:
1Without attention:
2 Encoder → [context] → Decoder
3 Long sentences lose information!
4
5With attention:
6 Encoder → [h_1, h_2, ..., h_n]
7 ↓ ↓ ↓
8 [attention weights]
9 ↓
10 Decoder (attends to relevant encoder states)Benefits:
1. Machine Translation
1English → French, Chinese, Spanish, ...
2Input: "How are you?"
3Output: "Comment allez-vous?"2. Text Summarization
1Long document → Short summary
2Input: [1000-word article]
3Output: [50-word summary]3. Conversational AI
1User message → Bot response
2Input: "What's the weather today?"
3Output: "It's sunny with a high of 75°F"4. Code Generation
1Natural language → Code
2Input: "Sort list in descending order"
3Output: "sorted(lst, reverse=True)"5. Time Series Forecasting
1Historical data → Future predictions
2Input: [last 30 days of stock prices]
3Output: [next 7 days forecast]1. Fixed-Size Bottleneck
2. Exposure Bias
3. Slow Inference
4. Repetition/Omission
Transformer Architecture has largely replaced LSTM seq2seq:
But LSTM seq2seq still useful for:
Seq2seq opened the door to modern NLP—paving the way for GPT, BERT, and Transformers!
Build a complete LSTM model that learns to generate text character-by-character. Train on Shakespeare text to generate new "Shakespeare-style" prose, demonstrating how RNNs learn patterns, grammar, and style from sequential data.
1# Character-Level Text Generation with LSTM
2# Learn to write like Shakespeare!
3
4import numpy as np
5import tensorflow as tf
6from tensorflow import keras
7from tensorflow.keras import layers
8import matplotlib.pyplot as plt
9import random
10
11print("=" * 80)
12print("🎭 CHARACTER-LEVEL TEXT GENERATION WITH LSTM")
13print("=" * 80)
14print()
15
16# ============================================================================
17# STEP 1: Load and Prepare Shakespeare Text
18# ============================================================================
19
20print("STEP 1: Loading Shakespeare dataset")
21print("-" * 80)
22
23# Download Shakespeare text
24path = keras.utils.get_file(
25 'shakespeare.txt',
26 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt'
27)
28
29# Read text
30with open(path, 'r', encoding='utf-8') as f:
31 text = f.read()
32
33print(f"Text length: {len(text):,} characters")
34print(f"\nFirst 200 characters:")
35print(text[:200])
36print()
37
38# Create character vocabulary
39chars = sorted(list(set(text)))
40vocab_size = len(chars)
41print(f"Vocabulary size: {vocab_size} unique characters")
42print(f"Characters: {''.join(chars[:50])}...")
43print()
44
45# Create character mappings
46char_to_idx = {ch: i for i, ch in enumerate(chars)}
47idx_to_char = {i: ch for i, ch in enumerate(chars)}
48
49# ============================================================================
50# STEP 2: Create Training Sequences
51# ============================================================================
52
53print("STEP 2: Creating training sequences")
54print("-" * 80)
55
56# Hyperparameters
57seq_length = 40 # Input sequence length
58step = 3 # Step size for creating sequences
59
60# Create sequences
61sequences = []
62next_chars = []
63
64for i in range(0, len(text) - seq_length, step):
65 sequences.append(text[i:i + seq_length])
66 next_chars.append(text[i + seq_length])
67
68print(f"Number of sequences: {len(sequences):,}")
69print(f"Sequence length: {seq_length}")
70print()
71
72# Show examples
73print("Example sequences:")
74for i in range(3):
75 print(f" Input: '{sequences[i]}'")
76 print(f" Target: '{next_chars[i]}'")
77 print()
78
79# Vectorize sequences
80print("Vectorizing sequences...")
81X = np.zeros((len(sequences), seq_length, vocab_size), dtype=bool)
82y = np.zeros((len(sequences), vocab_size), dtype=bool)
83
84for i, sequence in enumerate(sequences):
85 for t, char in enumerate(sequence):
86 X[i, t, char_to_idx[char]] = 1
87 y[i, char_to_idx[next_chars[i]]] = 1
88
89print(f"X shape: {X.shape} (samples, sequence_length, vocab_size)")
90print(f"y shape: {y.shape} (samples, vocab_size)")
91print()
92
93# ============================================================================
94# STEP 3: Build LSTM Model
95# ============================================================================
96
97print("STEP 3: Building LSTM model")
98print("-" * 80)
99
100model = keras.Sequential([
101 # LSTM layer 1
102 layers.LSTM(
103 128,
104 return_sequences=True,
105 input_shape=(seq_length, vocab_size),
106 dropout=0.2,
107 recurrent_dropout=0.2,
108 name='lstm1'
109 ),
110
111 # LSTM layer 2
112 layers.LSTM(
113 128,
114 return_sequences=False,
115 dropout=0.2,
116 recurrent_dropout=0.2,
117 name='lstm2'
118 ),
119
120 # Output layer (next character probability distribution)
121 layers.Dense(vocab_size, activation='softmax', name='output')
122], name='shakespeare_lstm')
123
124# Compile
125model.compile(
126 optimizer=keras.optimizers.RMSprop(learning_rate=0.01),
127 loss='categorical_crossentropy',
128 metrics=['accuracy']
129)
130
131print("Model architecture:")
132model.summary()
133print()
134
135# ============================================================================
136# STEP 4: Training with Custom Callback
137# ============================================================================
138
139print("STEP 4: Training LSTM")
140print("-" * 80)
141
142# Text generation callback (generates samples during training)
143class TextGenerationCallback(keras.callbacks.Callback):
144 def __init__(self, seed_text, temperature=0.5):
145 self.seed_text = seed_text
146 self.temperature = temperature
147
148 def on_epoch_end(self, epoch, logs=None):
149 print(f"\n--- Generating text after Epoch {epoch + 1} ---")
150 generated = self.generate_text(self.seed_text, 200)
151 print(generated)
152 print()
153
154 def generate_text(self, seed_text, length=100):
155 """Generate text using current model"""
156 generated = seed_text
157
158 for _ in range(length):
159 # Prepare input
160 x_pred = np.zeros((1, seq_length, vocab_size))
161 for t, char in enumerate(seed_text):
162 if char in char_to_idx:
163 x_pred[0, t, char_to_idx[char]] = 1
164
165 # Predict next character
166 preds = self.model.predict(x_pred, verbose=0)[0]
167 next_index = self.sample(preds, self.temperature)
168 next_char = idx_to_char[next_index]
169
170 generated += next_char
171 seed_text = seed_text[1:] + next_char
172
173 return generated
174
175 def sample(self, preds, temperature=1.0):
176 """Sample from probability distribution with temperature"""
177 preds = np.asarray(preds).astype('float64')
178 preds = np.log(preds + 1e-8) / temperature
179 exp_preds = np.exp(preds)
180 preds = exp_preds / np.sum(exp_preds)
181 probas = np.random.multinomial(1, preds, 1)
182 return np.argmax(probas)
183
184# Seed text for generation
185seed_text = "ROMEO:\nOh, wherefore art thou, "
186
187# Train model
188print("Training started...")
189print("(Sample text will be generated after each epoch)")
190print()
191
192history = model.fit(
193 X, y,
194 batch_size=128,
195 epochs=10,
196 validation_split=0.1,
197 callbacks=[
198 TextGenerationCallback(seed_text, temperature=0.5),
199 keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.5, patience=1)
200 ],
201 verbose=1
202)
203
204print("\nTraining complete!")
205print()
206
207# ============================================================================
208# STEP 5: Generate Text with Different Temperatures
209# ============================================================================
210
211print("STEP 5: Generating text with different temperatures")
212print("-" * 80)
213print()
214
215def generate_text(model, seed_text, length=500, temperature=0.5):
216 """
217 Generate text using trained model
218
219 Temperature:
220 - Low (0.2-0.5): Conservative, more predictable
221 - Medium (0.7-1.0): Balanced creativity
222 - High (1.2-2.0): Creative, more random
223 """
224 generated = seed_text
225
226 for _ in range(length):
227 x_pred = np.zeros((1, seq_length, vocab_size))
228
229 # Prepare input (last seq_length characters)
230 input_seq = generated[-seq_length:]
231 for t, char in enumerate(input_seq):
232 if char in char_to_idx:
233 x_pred[0, t, char_to_idx[char]] = 1
234
235 # Predict
236 preds = model.predict(x_pred, verbose=0)[0]
237
238 # Sample with temperature
239 preds = np.asarray(preds).astype('float64')
240 preds = np.log(preds + 1e-8) / temperature
241 exp_preds = np.exp(preds)
242 preds = exp_preds / np.sum(exp_preds)
243 probas = np.random.multinomial(1, preds, 1)
244 next_index = np.argmax(probas)
245
246 next_char = idx_to_char[next_index]
247 generated += next_char
248
249 return generated
250
251# Generate with different temperatures
252temperatures = [0.2, 0.5, 1.0, 1.5]
253seed = "HAMLET:\nTo be, or not to be, that "
254
255for temp in temperatures:
256 print(f"Temperature = {temp}")
257 print("=" * 70)
258 generated = generate_text(model, seed, length=300, temperature=temp)
259 print(generated)
260 print("\n")
261
262# ============================================================================
263# STEP 6: Visualize Training History
264# ============================================================================
265
266print("STEP 6: Training visualization")
267print("-" * 80)
268
269fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
270
271# Loss
272epochs = range(1, len(history.history['loss']) + 1)
273ax1.plot(epochs, history.history['loss'], 'b-', linewidth=2, label='Training Loss')
274ax1.plot(epochs, history.history['val_loss'], 'r-', linewidth=2, label='Validation Loss')
275ax1.set_xlabel('Epoch', fontweight='bold', fontsize=12)
276ax1.set_ylabel('Loss', fontweight='bold', fontsize=12)
277ax1.set_title('Model Loss', fontweight='bold', fontsize=14)
278ax1.legend(fontsize=11)
279ax1.grid(True, alpha=0.3)
280
281# Accuracy
282ax2.plot(epochs, history.history['accuracy'], 'b-', linewidth=2, label='Training Accuracy')
283ax2.plot(epochs, history.history['val_accuracy'], 'r-', linewidth=2, label='Validation Accuracy')
284ax2.set_xlabel('Epoch', fontweight='bold', fontsize=12)
285ax2.set_ylabel('Accuracy', fontweight='bold', fontsize=12)
286ax2.set_title('Model Accuracy', fontweight='bold', fontsize=14)
287ax2.legend(fontsize=11)
288ax2.grid(True, alpha=0.3)
289
290plt.tight_layout()
291plt.show()
292
293# ============================================================================
294# KEY INSIGHTS
295# ============================================================================
296
297print("\n" + "=" * 80)
298print("🎓 KEY INSIGHTS: CHARACTER-LEVEL TEXT GENERATION")
299print("=" * 80)
300print()
301
302print("1️⃣ LSTM LEARNS PATTERNS")
303print(" • Spelling: Learns common letter combinations")
304print(" • Grammar: Learns sentence structure")
305print(" • Style: Mimics Shakespeare's writing style")
306print(" • Context: Maintains coherent topics")
307print()
308
309print("2️⃣ TEMPERATURE CONTROLS CREATIVITY")
310print(" • Low (0.2-0.5): Safe, predictable, repetitive")
311print(" • Medium (0.7-1.0): Balanced, most coherent")
312print(" • High (1.5+): Creative but less coherent")
313print()
314
315print("3️⃣ SEQUENCE LENGTH MATTERS")
316print(" • Short (10-20): Only captures local patterns")
317print(" • Medium (40-60): Good balance (used here)")
318print(" • Long (100+): Better context, more GPU memory")
319print()
320
321print("4️⃣ TRAINING DYNAMICS")
322print(f" • Final training accuracy: {history.history['accuracy'][-1]*100:.1f}%")
323print(f" • Final validation accuracy: {history.history['val_accuracy'][-1]*100:.1f}%")
324print(" • Model predicts next character from context")
325print()
326
327print("5️⃣ LIMITATIONS")
328print(" • Character-level: Slow, doesn't understand words")
329print(" • Long-term coherence: Loses plot after ~100 chars")
330print(" • No semantic understanding: Just pattern matching")
331print()
332
333print("=" * 80)
334print("🚀 NEXT STEPS")
335print("=" * 80)
336print(" • Try word-level generation (better coherence)")
337print(" • Add GRU instead of LSTM (faster training)")
338print(" • Implement attention mechanism (better context)")
339print(" • Train longer for better quality")
340print(" • Try different datasets (code, poetry, etc.)")
341print()
342print("Modern alternative: GPT, BERT, Transformers →")This comprehensive example demonstrates how LSTMs learn to generate text by capturing patterns in sequential data.
1. Complete Pipeline
The example walks through every step:
2. Character-Level Learning
Why character-level instead of word-level?
Trade-off: Slower and less coherent than word-level models, but more flexible.
3. Temperature Sampling
Temperature controls randomness in generation:
1# Without temperature (argmax):
2next_char = chars[np.argmax(predictions)]
3# → Always picks most likely character (deterministic, boring)
4
5# With temperature (sampling):
6predictions = predictions ** (1/temperature)
7predictions = predictions / np.sum(predictions)
8next_char = np.random.choice(chars, p=predictions)
9# → Samples from distribution (creative, varied)Temperature=0.2: "ROMEO: I love thee, my dear Juliet, thou art so fair..."
Temperature=1.0: "ROMEO: I prithee, good sir, what manner of beast..."
Temperature=1.5: "ROMEO: Zounds! By my troth, hath yonder moon..."
4. Real-Time Generation During Training
The TextGenerationCallback shows how the model improves over epochs:
Epoch 1: "ROMEO: adlk ajd lkja dlkj ..." (random gibberish)
Epoch 3: "ROMEO: I am the to the the..." (words forming)
Epoch 7: "ROMEO: I love thee, my lady..." (coherent!)
Epoch 10: "ROMEO: But soft! What light through yonder window breaks?" (Shakespeare-like!)
5. Training Insights
Accuracy Interpretation:
Validation Loss:
Layer 1 (early LSTM):
Layer 2 (deep LSTM):
Output Layer:
Better Architecture:
1# Add more layers
2model = Sequential([
3 LSTM(256, return_sequences=True),
4 Dropout(0.3),
5 LSTM(256, return_sequences=True),
6 Dropout(0.3),
7 LSTM(128),
8 Dense(vocab_size, activation='softmax')
9])Word-Level Generation:
1# Use word embeddings instead of character one-hot
2Embedding(vocab_size, 256) → LSTM → Dense
3# Faster, more coherent, but needs larger vocabularyBidirectional LSTM:
1Bidirectional(LSTM(128, return_sequences=True))
2# Better for tasks with full context (not generation)This exact technique powers:
LSTM Limitations:
Modern Alternatives (2025):
But LSTMs still used for:
This example is your foundation for understanding how neural networks process and generate sequential data!
Outstanding! You've mastered Recurrent Neural Networks—from vanilla RNNs and the vanishing gradient problem to LSTM gates and GRU efficiency. You built a character-level text generator that learned Shakespeare's writing style purely from sequences!
Next: In Lesson 10: Advanced RNN Architectures, we'll explore cutting-edge techniques:
From basic recurrence to the attention revolution—let's build the architectures that power modern NLP! 🚀
Your progress is saved in this browser, on this device only.