Master neural language models for text generation, from n-gram models to GPT-style transformers with various decoding strategies.
Language generation is the task of producing coherent, contextual text given a prompt or condition. This capability powers chatbots, story generators, code completion, and content creation tools. Modern language models like GPT learn to predict the next token, then generate text autoregressively.
The quality of generated text depends not only on the model but also on the decoding strategy. Greedy decoding, beam search, and various sampling methods offer different trade-offs between quality, diversity, and creativity.
In this lesson, you'll understand how language models generate text and implement different decoding strategies.
Language models predict the probability of the next token given context.
Autoregressive Generation:
1class LanguageModel:
2 """Simple autoregressive language model"""
3
4 def __init__(self, vocab_size, d_model):
5 self.embedding = nn.Embedding(vocab_size, d_model)
6 self.transformer = TransformerDecoder(d_model, n_layers=6)
7 self.output = nn.Linear(d_model, vocab_size)
8
9 def forward(self, x):
10 # x: (batch, seq_len)
11 embedded = self.embedding(x)
12 hidden = self.transformer(embedded)
13 logits = self.output(hidden)
14 return logits # (batch, seq_len, vocab_size)
15
16 def generate(self, prompt_ids, max_length=50):
17 generated = prompt_ids.clone()
18
19 for _ in range(max_length):
20 logits = self.forward(generated)
21 next_token_logits = logits[:, -1, :] # Last position
22 next_token = torch.argmax(next_token_logits, dim=-1)
23 generated = torch.cat([generated, next_token.unsqueeze(1)], dim=1)
24
25 if next_token == eos_token_id:
26 break
27
28 return generatedEach token is predicted based on all previous tokens.
Different strategies for selecting the next token:
Greedy Decoding:
1def greedy_decode(logits):
2 """Always pick highest probability token"""
3 return torch.argmax(logits, dim=-1)Temperature Sampling:
1def temperature_sample(logits, temperature=1.0):
2 """Sample with temperature scaling"""
3 scaled_logits = logits / temperature
4 probs = F.softmax(scaled_logits, dim=-1)
5 return torch.multinomial(probs, num_samples=1)Top-k Sampling:
1def top_k_sample(logits, k=50):
2 """Sample from top-k tokens only"""
3 top_k_logits, top_k_indices = torch.topk(logits, k)
4 probs = F.softmax(top_k_logits, dim=-1)
5 sampled_idx = torch.multinomial(probs, num_samples=1)
6 return top_k_indices[sampled_idx]Top-p (Nucleus) Sampling:
1def top_p_sample(logits, p=0.9):
2 """Sample from smallest set with cumulative prob >= p"""
3 sorted_logits, sorted_indices = torch.sort(logits, descending=True)
4 cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
5
6 # Remove tokens with cumulative probability above threshold
7 sorted_indices_to_remove = cumulative_probs > p
8 sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
9 sorted_indices_to_remove[..., 0] = 0
10
11 indices_to_remove = sorted_indices[sorted_indices_to_remove]
12 logits[indices_to_remove] = -float('Inf')
13
14 probs = F.softmax(logits, dim=-1)
15 return torch.multinomial(probs, num_samples=1)Techniques for controlling generated text:
Repetition Penalty:
1def apply_repetition_penalty(logits, generated_ids, penalty=1.2):
2 """Reduce probability of already generated tokens"""
3 for token_id in set(generated_ids):
4 logits[token_id] /= penalty
5 return logitsLength Penalty:
1def length_normalized_score(log_prob, length, alpha=0.6):
2 """Normalize scores by length to avoid short sequences"""
3 return log_prob / (length ** alpha)Constrained Generation:
1def constrained_decode(logits, must_include, must_exclude):
2 """Force or block specific tokens"""
3 for token_id in must_exclude:
4 logits[token_id] = -float('Inf')
5
6 # Boost must-include tokens if not yet generated
7 for token_id in must_include:
8 logits[token_id] += 10.0
9
10 return logitsThese controls help produce more coherent, diverse, and task-appropriate text.
Effective prompts guide generation quality:
1# Few-shot prompting
2few_shot_prompt = """
3Translate English to French:
4
5English: Hello, how are you?
6French: Bonjour, comment allez-vous?
7
8English: The weather is nice today.
9French: Le temps est beau aujourd'hui.
10
11English: I love programming.
12French:"""
13
14# Instruction-style prompting
15instruction_prompt = """
16Write a short poem about autumn leaves.
17Style: Haiku format (5-7-5 syllables)
18Tone: Melancholic but beautiful
19
20Poem:
21"""
22
23# Chain-of-thought prompting
24cot_prompt = """
25Solve this math problem step by step:
26
27Problem: If a train travels at 60 mph for 2.5 hours, how far does it go?
28
29Solution:
30Step 1: Identify the formula (distance = speed × time)
31Step 2: Plug in the values (distance = 60 × 2.5)
32Step 3: Calculate (distance = 150 miles)
33
34Problem: A car uses 3 gallons of gas per 100 miles. How much gas for 250 miles?
35
36Solution:
37"""Good prompts provide context, examples, and clear instructions.
Implement a complete text generation system with multiple decoding strategies.
1import numpy as np
2from collections import Counter
3import re
4
5class SimpleLanguageModel:
6 """N-gram language model with various decoding strategies"""
7
8 def __init__(self, n=3):
9 self.n = n
10 self.ngram_counts = Counter()
11 self.context_counts = Counter()
12 self.vocab = set()
13
14 def tokenize(self, text):
15 """Simple tokenization"""
16 text = text.lower()
17 tokens = re.findall(r'\b\w+\b|[.!?]', text)
18 return tokens
19
20 def train(self, text):
21 """Train on text corpus"""
22 tokens = self.tokenize(text)
23 self.vocab = set(tokens)
24
25 # Count n-grams
26 for i in range(len(tokens) - self.n + 1):
27 context = tuple(tokens[i:i + self.n - 1])
28 target = tokens[i + self.n - 1]
29 self.ngram_counts[(context, target)] += 1
30 self.context_counts[context] += 1
31
32 print(f"Trained on {len(tokens)} tokens")
33 print(f"Vocabulary size: {len(self.vocab)}")
34 print(f"Unique contexts: {len(self.context_counts)}")
35
36 def get_next_token_probs(self, context):
37 """Get probability distribution over next tokens"""
38 context = tuple(context[-(self.n - 1):])
39 total = self.context_counts.get(context, 0)
40
41 if total == 0:
42 # Uniform distribution if context not seen
43 return {word: 1.0 / len(self.vocab) for word in self.vocab}
44
45 probs = {}
46 for word in self.vocab:
47 count = self.ngram_counts.get((context, word), 0)
48 probs[word] = (count + 0.1) / (total + 0.1 * len(self.vocab)) # Smoothing
49
50 return probs
51
52 def greedy_generate(self, prompt, max_length=20):
53 """Greedy decoding - always pick most likely"""
54 tokens = self.tokenize(prompt)
55
56 for _ in range(max_length):
57 probs = self.get_next_token_probs(tokens)
58 next_token = max(probs, key=probs.get)
59 tokens.append(next_token)
60
61 if next_token in '.!?':
62 break
63
64 return ' '.join(tokens)
65
66 def sample_generate(self, prompt, max_length=20, temperature=1.0):
67 """Temperature sampling"""
68 tokens = self.tokenize(prompt)
69
70 for _ in range(max_length):
71 probs = self.get_next_token_probs(tokens)
72
73 # Apply temperature
74 words = list(probs.keys())
75 log_probs = np.array([np.log(probs[w] + 1e-10) for w in words])
76 scaled = log_probs / temperature
77 exp_scaled = np.exp(scaled - np.max(scaled))
78 prob_dist = exp_scaled / exp_scaled.sum()
79
80 next_token = np.random.choice(words, p=prob_dist)
81 tokens.append(next_token)
82
83 if next_token in '.!?':
84 break
85
86 return ' '.join(tokens)
87
88 def top_k_generate(self, prompt, max_length=20, k=5, temperature=0.8):
89 """Top-k sampling"""
90 tokens = self.tokenize(prompt)
91
92 for _ in range(max_length):
93 probs = self.get_next_token_probs(tokens)
94
95 # Get top-k tokens
96 sorted_probs = sorted(probs.items(), key=lambda x: x[1], reverse=True)
97 top_k = sorted_probs[:k]
98
99 # Sample from top-k with temperature
100 words = [w for w, _ in top_k]
101 log_probs = np.array([np.log(p + 1e-10) for _, p in top_k])
102 scaled = log_probs / temperature
103 exp_scaled = np.exp(scaled - np.max(scaled))
104 prob_dist = exp_scaled / exp_scaled.sum()
105
106 next_token = np.random.choice(words, p=prob_dist)
107 tokens.append(next_token)
108
109 if next_token in '.!?':
110 break
111
112 return ' '.join(tokens)
113
114# Training corpus
115corpus = """
116The sun rises in the east and sets in the west. The moon shines at night.
117Birds fly in the sky. Fish swim in the sea. Trees grow in the forest.
118The wind blows through the mountains. Rivers flow to the ocean.
119Flowers bloom in spring. Leaves fall in autumn. Snow falls in winter.
120The cat sleeps on the mat. The dog runs in the park.
121Children play in the garden. Stars twinkle in the night sky.
122The sun warms the earth. Rain falls from the clouds.
123"""
124
125# Train model
126lm = SimpleLanguageModel(n=3)
127lm.train(corpus)
128
129print("\nLANGUAGE GENERATION")
130print("=" * 60)
131
132prompts = ["the sun", "the cat", "birds fly"]
133
134for prompt in prompts:
135 print(f"\nPrompt: '{prompt}'")
136 print("-" * 40)
137
138 print("Greedy:", lm.greedy_generate(prompt, max_length=10))
139 print("Sample (T=1.0):", lm.sample_generate(prompt, max_length=10, temperature=1.0))
140 print("Sample (T=0.5):", lm.sample_generate(prompt, max_length=10, temperature=0.5))
141 print("Top-5:", lm.top_k_generate(prompt, max_length=10, k=5))
142
143print("\n" + "=" * 60)
144print("TEMPERATURE COMPARISON")
145print("-" * 60)
146
147prompt = "the moon"
148for temp in [0.3, 0.7, 1.0, 1.5]:
149 print(f"T={temp}: {lm.sample_generate(prompt, max_length=12, temperature=temp)}")This demonstrates a simple n-gram language model with multiple decoding strategies including greedy, temperature sampling, and top-k sampling.
In our next lesson, "Multilingual NLP," you'll learn to build systems that work across multiple languages using cross-lingual models and techniques.
Your progress is saved in this browser, on this device only.