Learn extractive and abstractive summarization techniques to automatically condense documents while preserving essential information.
Text summarization automatically creates concise versions of longer documents while preserving key information. This capability is essential for news aggregation, document management, research assistance, and any application dealing with information overload.
Two main approaches exist: extractive summarization selects and combines existing sentences from the source, while abstractive summarization generates new text that may paraphrase or rephrase content. Modern transformer models excel at abstractive summarization, producing human-like summaries.
In this lesson, you'll implement both approaches and learn to evaluate summary quality.
Extractive methods select the most important sentences from the source.
Sentence Scoring Methods:
1def score_sentences(document):
2 """Score sentences for extraction"""
3 sentences = sent_tokenize(document)
4
5 # Method 1: TF-IDF based
6 tfidf = TfidfVectorizer()
7 tfidf_matrix = tfidf.fit_transform(sentences)
8 sentence_scores = tfidf_matrix.sum(axis=1).flatten()
9
10 # Method 2: Position-based (first sentences are important)
11 position_scores = [1.0 / (i + 1) for i in range(len(sentences))]
12
13 # Method 3: Length normalization (avoid very short/long)
14 length_scores = [min(len(s.split()) / 20, 1) for s in sentences]
15
16 # Combine scores
17 combined = []
18 for i in range(len(sentences)):
19 score = (0.5 * sentence_scores[i] +
20 0.3 * position_scores[i] +
21 0.2 * length_scores[i])
22 combined.append((score, i, sentences[i]))
23
24 return sorted(combined, reverse=True)Select top-k sentences and reorder by original position.
TextRank applies PageRank to sentence similarity graphs.
1import networkx as nx
2from sklearn.metrics.pairwise import cosine_similarity
3
4def textrank_summarize(sentences, embeddings, top_n=3):
5 """TextRank-based extractive summarization"""
6 # Build similarity graph
7 similarity_matrix = cosine_similarity(embeddings)
8
9 # Create graph
10 G = nx.from_numpy_array(similarity_matrix)
11
12 # Apply PageRank
13 scores = nx.pagerank(G)
14
15 # Rank sentences
16 ranked = sorted(
17 [(scores[i], i, sentences[i]) for i in range(len(sentences))],
18 reverse=True
19 )
20
21 # Select top sentences and reorder by position
22 top_sentences = sorted(ranked[:top_n], key=lambda x: x[1])
23
24 return ' '.join([s[2] for s in top_sentences])TextRank identifies central sentences in the document's semantic space.
Abstractive methods generate new text using seq2seq models.
1from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM
2
3class AbstractiveSummarizer:
4 """T5/BART based abstractive summarizer"""
5
6 def __init__(self, model_name='t5-small'):
7 self.tokenizer = AutoTokenizer.from_pretrained(model_name)
8 self.model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
9
10 def summarize(self, text, max_length=150, min_length=40):
11 # T5 requires prefix
12 if 't5' in model_name:
13 text = "summarize: " + text
14
15 inputs = self.tokenizer(
16 text,
17 return_tensors='pt',
18 max_length=512,
19 truncation=True
20 )
21
22 summary_ids = self.model.generate(
23 inputs['input_ids'],
24 max_length=max_length,
25 min_length=min_length,
26 num_beams=4,
27 length_penalty=2.0,
28 early_stopping=True
29 )
30
31 return self.tokenizer.decode(summary_ids[0], skip_special_tokens=True)
32
33# Using pipeline (simpler)
34summarizer = pipeline('summarization', model='facebook/bart-large-cnn')
35summary = summarizer(article, max_length=130, min_length=30)[0]['summary_text']BART and T5 are pre-trained on summarization datasets.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures summary quality.
1from collections import Counter
2
3def rouge_n(reference, candidate, n=1):
4 """Compute ROUGE-N score"""
5 def get_ngrams(text, n):
6 tokens = text.lower().split()
7 return [tuple(tokens[i:i+n]) for i in range(len(tokens)-n+1)]
8
9 ref_ngrams = Counter(get_ngrams(reference, n))
10 cand_ngrams = Counter(get_ngrams(candidate, n))
11
12 # Overlap
13 overlap = sum((ref_ngrams & cand_ngrams).values())
14
15 # Recall (most important for ROUGE)
16 recall = overlap / sum(ref_ngrams.values()) if ref_ngrams else 0
17
18 # Precision
19 precision = overlap / sum(cand_ngrams.values()) if cand_ngrams else 0
20
21 # F1
22 f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
23
24 return {'precision': precision, 'recall': recall, 'f1': f1}
25
26def rouge_l(reference, candidate):
27 """Compute ROUGE-L (Longest Common Subsequence)"""
28 ref_tokens = reference.lower().split()
29 cand_tokens = candidate.lower().split()
30
31 # LCS dynamic programming
32 m, n = len(ref_tokens), len(cand_tokens)
33 dp = [[0] * (n + 1) for _ in range(m + 1)]
34
35 for i in range(1, m + 1):
36 for j in range(1, n + 1):
37 if ref_tokens[i-1] == cand_tokens[j-1]:
38 dp[i][j] = dp[i-1][j-1] + 1
39 else:
40 dp[i][j] = max(dp[i-1][j], dp[i][j-1])
41
42 lcs_length = dp[m][n]
43 recall = lcs_length / m if m > 0 else 0
44 precision = lcs_length / n if n > 0 else 0
45 f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
46
47 return {'precision': precision, 'recall': recall, 'f1': f1}Implement a complete extractive summarization system with multiple scoring methods.
1import numpy as np
2from collections import Counter
3import re
4
5class ExtractiveSummarizer:
6 """Multi-method extractive summarization"""
7
8 def __init__(self):
9 self.idf = {}
10
11 def preprocess(self, text):
12 """Clean and tokenize"""
13 text = text.lower()
14 text = re.sub(r'[^\w\s]', '', text)
15 return text.split()
16
17 def sent_tokenize(self, text):
18 """Split text into sentences"""
19 sentences = re.split(r'(?<=[.!?])\s+', text)
20 return [s.strip() for s in sentences if len(s.strip()) > 10]
21
22 def compute_tfidf(self, sentences):
23 """Compute TF-IDF scores for sentences"""
24 # Build vocabulary
25 doc_tokens = [self.preprocess(s) for s in sentences]
26 all_words = set(w for tokens in doc_tokens for w in tokens)
27
28 # IDF
29 n_docs = len(sentences)
30 doc_freq = Counter()
31 for tokens in doc_tokens:
32 for word in set(tokens):
33 doc_freq[word] += 1
34
35 self.idf = {w: np.log((n_docs + 1) / (df + 1)) + 1 for w, df in doc_freq.items()}
36
37 # TF-IDF scores per sentence
38 scores = []
39 for tokens in doc_tokens:
40 tf = Counter(tokens)
41 score = sum(tf[w] * self.idf.get(w, 0) for w in tf)
42 scores.append(score / (len(tokens) + 1)) # Length normalize
43
44 return np.array(scores)
45
46 def compute_position_scores(self, n_sentences):
47 """First and last sentences are often important"""
48 scores = []
49 for i in range(n_sentences):
50 # High score for beginning, moderate for end
51 if i < 3:
52 scores.append(1.0 - i * 0.15)
53 elif i >= n_sentences - 2:
54 scores.append(0.5)
55 else:
56 scores.append(0.3)
57 return np.array(scores)
58
59 def compute_keyword_scores(self, sentences, keywords):
60 """Score sentences by keyword overlap"""
61 keyword_set = set(w.lower() for w in keywords)
62 scores = []
63
64 for sentence in sentences:
65 words = set(self.preprocess(sentence))
66 overlap = len(words & keyword_set)
67 scores.append(overlap / (len(keyword_set) + 1))
68
69 return np.array(scores)
70
71 def summarize(self, text, n_sentences=3, keywords=None):
72 """Generate extractive summary"""
73 sentences = self.sent_tokenize(text)
74
75 if len(sentences) <= n_sentences:
76 return text
77
78 # Compute different scores
79 tfidf_scores = self.compute_tfidf(sentences)
80 position_scores = self.compute_position_scores(len(sentences))
81
82 # Normalize scores
83 tfidf_scores = tfidf_scores / (tfidf_scores.max() + 1e-10)
84 position_scores = position_scores / (position_scores.max() + 1e-10)
85
86 # Combine scores
87 combined_scores = 0.6 * tfidf_scores + 0.4 * position_scores
88
89 # Add keyword scores if provided
90 if keywords:
91 keyword_scores = self.compute_keyword_scores(sentences, keywords)
92 keyword_scores = keyword_scores / (keyword_scores.max() + 1e-10)
93 combined_scores = 0.5 * combined_scores + 0.5 * keyword_scores
94
95 # Select top sentences
96 top_indices = np.argsort(combined_scores)[::-1][:n_sentences]
97
98 # Reorder by original position
99 top_indices = sorted(top_indices)
100
101 summary_sentences = [sentences[i] for i in top_indices]
102 return ' '.join(summary_sentences)
103
104# Test article
105article = """
106Artificial intelligence is transforming industries across the globe. Machine learning, a subset of AI, enables computers to learn from data without explicit programming. Deep learning, using neural networks with multiple layers, has achieved remarkable results in image recognition and natural language processing.
107
108Companies like Google, Microsoft, and OpenAI are investing billions in AI research. The technology has applications in healthcare, finance, transportation, and entertainment. Autonomous vehicles use AI to navigate roads safely. Medical AI helps doctors diagnose diseases earlier.
109
110However, AI also raises concerns about job displacement and algorithmic bias. Experts emphasize the need for ethical AI development and regulation. The future of AI depends on responsible innovation that benefits society while minimizing risks.
111
112Recent breakthroughs include large language models like GPT and image generators like DALL-E. These systems can write essays, code software, and create art. The rapid progress has sparked debates about AI safety and the potential for artificial general intelligence.
113"""
114
115# Create summarizer
116summarizer = ExtractiveSummarizer()
117
118print("EXTRACTIVE SUMMARIZATION")
119print("=" * 60)
120print("\nOriginal Article Length:", len(article.split()), "words")
121
122# Generate summaries
123print("\n" + "-" * 60)
124print("3-Sentence Summary:")
125print("-" * 60)
126summary_3 = summarizer.summarize(article, n_sentences=3)
127print(summary_3)
128
129print("\n" + "-" * 60)
130print("Keyword-Focused Summary (keywords: AI, healthcare, safety):")
131print("-" * 60)
132summary_kw = summarizer.summarize(article, n_sentences=3, keywords=['AI', 'healthcare', 'safety'])
133print(summary_kw)
134
135print("\n" + "-" * 60)
136print("Summary Length:", len(summary_3.split()), "words")
137print("Compression Ratio:", f"{len(summary_3.split()) / len(article.split()):.1%}")This demonstrates extractive summarization combining TF-IDF scoring, position weighting, and optional keyword focus to select the most important sentences.
In our next lesson, "Language Generation," you'll learn to build systems that generate coherent, contextual text using neural language models.
Your progress is saved in this browser, on this device only.