Understand the transformer architecture and BERT that revolutionized NLP with self-attention and pre-training on massive text corpora.
The Transformer architecture, introduced in "Attention Is All You Need" (2017), replaced recurrence with self-attention, enabling parallel processing and capturing long-range dependencies. BERT (Bidirectional Encoder Representations from Transformers) then showed that pre-training on large corpora produces universal representations that transfer to many NLP tasks.
These models achieve state-of-the-art results across NLP by learning contextual word representations where the same word gets different embeddings based on surrounding context - "bank" means different things in "river bank" vs "bank account".
In this lesson, you'll understand the core mechanisms and learn to apply pre-trained transformers to your tasks.
Self-attention computes relationships between all positions in a sequence.
Query, Key, Value:
1def self_attention(X, Wq, Wk, Wv):
2 """
3 X: input embeddings (seq_len, d_model)
4 Wq, Wk, Wv: projection matrices
5 """
6 Q = X @ Wq # Queries
7 K = X @ Wk # Keys
8 V = X @ Wv # Values
9
10 # Attention scores
11 d_k = K.shape[-1]
12 scores = (Q @ K.T) / np.sqrt(d_k) # Scaled dot-product
13 weights = softmax(scores, axis=-1)
14
15 # Weighted sum of values
16 output = weights @ V
17 return output, weightsEach position attends to all positions, capturing dependencies regardless of distance.
Multiple attention heads capture different relationship types.
1def multi_head_attention(X, heads=8, d_model=512):
2 """Multiple parallel attention heads"""
3 d_k = d_model // heads
4 outputs = []
5
6 for h in range(heads):
7 # Each head has its own projections
8 Wq = np.random.randn(d_model, d_k)
9 Wk = np.random.randn(d_model, d_k)
10 Wv = np.random.randn(d_model, d_k)
11
12 head_output, _ = self_attention(X, Wq, Wk, Wv)
13 outputs.append(head_output)
14
15 # Concatenate heads and project
16 concat = np.concatenate(outputs, axis=-1)
17 Wo = np.random.randn(d_model, d_model)
18 return concat @ WoDifferent heads learn: syntactic relations, semantic similarity, coreference, etc.
BERT learns bidirectional context through two pre-training tasks:
1. Masked Language Model (MLM):
2. Next Sentence Prediction (NSP):
1# MLM Example
2original = "The cat sat on the mat"
3masked = "The [MASK] sat on the [MASK]"
4# Model predicts: cat, mat
5
6# NSP Example
7sentence_a = "The weather is nice today."
8sentence_b_real = "Let's go for a walk." # IsNext
9sentence_b_fake = "I love programming." # NotNextAdapting pre-trained BERT for specific tasks:
1from transformers import BertForSequenceClassification, BertTokenizer
2
3# Load pre-trained model
4tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
5model = BertForSequenceClassification.from_pretrained(
6 'bert-base-uncased',
7 num_labels=2 # Binary classification
8)
9
10# Prepare data
11inputs = tokenizer(
12 "This movie was amazing!",
13 return_tensors="pt",
14 padding=True,
15 truncation=True
16)
17
18# Fine-tune (simplified)
19optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)
20
21for batch in train_dataloader:
22 outputs = model(**batch)
23 loss = outputs.loss
24 loss.backward()
25 optimizer.step()Build self-attention mechanism from scratch to understand transformers.
1import numpy as np
2
3def softmax(x, axis=-1):
4 exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
5 return exp_x / exp_x.sum(axis=axis, keepdims=True)
6
7class SelfAttention:
8 """Single-head self-attention"""
9
10 def __init__(self, d_model, d_k):
11 np.random.seed(42)
12 self.d_k = d_k
13 self.Wq = np.random.randn(d_model, d_k) * 0.1
14 self.Wk = np.random.randn(d_model, d_k) * 0.1
15 self.Wv = np.random.randn(d_model, d_k) * 0.1
16
17 def forward(self, X):
18 """
19 X: (seq_len, d_model) - input embeddings
20 Returns: (seq_len, d_k) - attended output
21 """
22 # Linear projections
23 Q = X @ self.Wq # Queries
24 K = X @ self.Wk # Keys
25 V = X @ self.Wv # Values
26
27 # Scaled dot-product attention
28 scores = Q @ K.T / np.sqrt(self.d_k)
29 weights = softmax(scores)
30
31 # Weighted combination
32 output = weights @ V
33 return output, weights
34
35class MultiHeadAttention:
36 """Multi-head self-attention"""
37
38 def __init__(self, d_model, n_heads):
39 self.n_heads = n_heads
40 self.d_k = d_model // n_heads
41 self.heads = [SelfAttention(d_model, self.d_k) for _ in range(n_heads)]
42 self.Wo = np.random.randn(d_model, d_model) * 0.1
43
44 def forward(self, X):
45 # Run all heads
46 outputs = []
47 all_weights = []
48 for head in self.heads:
49 out, weights = head.forward(X)
50 outputs.append(out)
51 all_weights.append(weights)
52
53 # Concatenate and project
54 concat = np.concatenate(outputs, axis=-1)
55 output = concat @ self.Wo
56 return output, all_weights
57
58# Create word embeddings for a sentence
59words = ['The', 'cat', 'sat', 'on', 'the', 'mat']
60d_model = 16
61np.random.seed(42)
62embeddings = np.random.randn(len(words), d_model)
63
64print("SELF-ATTENTION MECHANISM")
65print("=" * 60)
66print(f"\nInput: {words}")
67print(f"Embedding shape: {embeddings.shape}")
68
69# Single-head attention
70print("\n1. SINGLE-HEAD ATTENTION")
71print("-" * 60)
72attn = SelfAttention(d_model, d_k=8)
73output, weights = attn.forward(embeddings)
74
75print(f"Output shape: {output.shape}")
76print("\nAttention weights (which words attend to which):")
77print(f"{'':6}", end='')
78for w in words:
79 print(f"{w:>8}", end='')
80print()
81for i, w in enumerate(words):
82 print(f"{w:6}", end='')
83 for j in range(len(words)):
84 print(f"{weights[i,j]:>8.2f}", end='')
85 print()
86
87# Multi-head attention
88print("\n2. MULTI-HEAD ATTENTION")
89print("-" * 60)
90mha = MultiHeadAttention(d_model, n_heads=4)
91output, head_weights = mha.forward(embeddings)
92
93print(f"Number of heads: {len(head_weights)}")
94print(f"Output shape: {output.shape}")
95
96# Show what each head focuses on
97print("\nHead specialization (max attention per word):")
98for h, weights in enumerate(head_weights):
99 focus = []
100 for i, w in enumerate(words):
101 max_attn = np.argmax(weights[i])
102 if max_attn != i: # Skip self-attention
103 focus.append(f"{w}->{words[max_attn]}")
104 print(f" Head {h}: {', '.join(focus[:3])}...")This demonstrates the self-attention mechanism that powers transformers, showing how words attend to other words in the sequence.
In our next lesson, "Question Answering Systems," you'll build QA systems using BERT and extractive/generative approaches.
Your progress is saved in this browser, on this device only.