Master essential text preprocessing techniques including tokenization, stemming, lemmatization, and text normalization to prepare raw text for NLP tasks.
Text preprocessing is the foundation of every successful NLP project. Raw text data is inherently messy—it contains inconsistent capitalization, punctuation variations, special characters, and noise that can confuse machine learning models. Proper preprocessing transforms this chaotic data into clean, standardized input that models can learn from effectively.
The preprocessing decisions you make significantly impact downstream performance. Over-aggressive preprocessing might remove important information (like capitalization in named entity recognition), while insufficient preprocessing leaves noise that hurts model accuracy. Understanding these tradeoffs is essential for building robust NLP systems.
In this comprehensive lesson, you'll master the core preprocessing techniques used in production NLP systems: tokenization, normalization, stemming, lemmatization, and stopword removal. You'll implement these techniques from scratch and with industry-standard libraries like NLTK and spaCy.
Tokenization is the process of breaking text into smaller units called tokens. This fundamental step converts raw strings into sequences that NLP models can process.
Types of Tokenization:
1. Word Tokenization:
Splits text into individual words. Handles punctuation, contractions, and special cases.
1import re
2from typing import List
3
4def word_tokenize_simple(text: str) -> List[str]:
5 """Simple word tokenization using regex"""
6 # Split on whitespace and punctuation
7 tokens = re.findall(r"\b\w+\b", text.lower())
8 return tokens
9
10# Example
11text = "Hello, world! How's it going? I'm doing great."
12tokens = word_tokenize_simple(text)
13print(f"Tokens: {tokens}")
14# Output: ['hello', 'world', 'how', 's', 'it', 'going', 'i', 'm', 'doing', 'great']2. Sentence Tokenization:
Splits text into sentences. Handles abbreviations, decimals, and edge cases.
1import re
2
3def sentence_tokenize(text: str) -> List[str]:
4 """Split text into sentences"""
5 # Handle common abbreviations
6 abbreviations = ['Mr.', 'Mrs.', 'Dr.', 'Prof.', 'Inc.', 'Ltd.']
7 for abbr in abbreviations:
8 text = text.replace(abbr, abbr.replace('.', '<DOT>'))
9
10 # Split on sentence-ending punctuation
11 sentences = re.split(r'(?<=[.!?])\s+', text)
12
13 # Restore abbreviations
14 sentences = [s.replace('<DOT>', '.') for s in sentences]
15 return sentences
16
17text = "Dr. Smith works at Inc. He's great! What do you think?"
18sentences = sentence_tokenize(text)
19print(f"Sentences: {sentences}")3. Subword Tokenization (Modern NLP):
Breaks words into smaller meaningful units. Used by BERT, GPT, and modern transformers.
1# Byte-Pair Encoding (BPE) concept
2def simple_bpe_example():
3 """Demonstrate BPE tokenization concept"""
4 # Vocabulary with subword units
5 vocab = ['un', 'happi', 'ness', 'happy', 'ly', 'ing']
6
7 # Words can be broken into subwords
8 examples = {
9 'unhappiness': ['un', 'happi', 'ness'],
10 'happily': ['happi', 'ly'],
11 'unhappy': ['un', 'happy']
12 }
13
14 for word, subwords in examples.items():
15 print(f"{word} → {' + '.join(subwords)}")
16
17simple_bpe_example()NLTK and spaCy Tokenization:
1import nltk
2import spacy
3
4# NLTK tokenization
5from nltk.tokenize import word_tokenize, sent_tokenize
6nltk.download('punkt', quiet=True)
7
8text = "Natural language processing is fascinating. It powers many AI applications!"
9print(f"NLTK words: {word_tokenize(text)}")
10print(f"NLTK sentences: {sent_tokenize(text)}")
11
12# spaCy tokenization (more sophisticated)
13nlp = spacy.load('en_core_web_sm')
14doc = nlp(text)
15print(f"spaCy tokens: {[token.text for token in doc]}")Key Considerations:
Text normalization transforms text into a consistent, standardized format. This reduces vocabulary size and helps models generalize better.
Common Normalization Techniques:
1. Lowercasing:
1def normalize_case(text: str, preserve_upper: bool = False) -> str:
2 """Convert to lowercase, optionally preserve all-caps"""
3 if preserve_upper:
4 # Preserve words that are all uppercase (acronyms)
5 words = text.split()
6 normalized = []
7 for word in words:
8 if word.isupper() and len(word) > 1:
9 normalized.append(word)
10 else:
11 normalized.append(word.lower())
12 return ' '.join(normalized)
13 return text.lower()
14
15text = "NASA announced a new mission to MARS. The CEO was excited!"
16print(f"Simple: {normalize_case(text)}")
17print(f"Preserve acronyms: {normalize_case(text, preserve_upper=True)}")2. Removing Punctuation and Special Characters:
1import re
2import string
3
4def remove_punctuation(text: str, keep_sentence_end: bool = False) -> str:
5 """Remove punctuation from text"""
6 if keep_sentence_end:
7 # Keep sentence-ending punctuation
8 return re.sub(r'[^\w\s.!?]', '', text)
9 # Remove all punctuation
10 return text.translate(str.maketrans('', '', string.punctuation))
11
12def remove_special_chars(text: str) -> str:
13 """Remove special characters and extra whitespace"""
14 # Remove URLs
15 text = re.sub(r'https?://\S+|www\.\S+', '', text)
16 # Remove emails
17 text = re.sub(r'\S+@\S+', '', text)
18 # Remove numbers (optional)
19 # text = re.sub(r'\d+', '', text)
20 # Remove extra whitespace
21 text = ' '.join(text.split())
22 return text
23
24text = "Check out https://example.com! Email me at [email protected]... Thanks!!!"
25print(f"Original: {text}")
26print(f"Cleaned: {remove_special_chars(text)}")3. Unicode Normalization:
1import unicodedata
2
3def normalize_unicode(text: str) -> str:
4 """Normalize Unicode characters"""
5 # NFKD normalization decomposes characters
6 text = unicodedata.normalize('NFKD', text)
7 # Remove non-ASCII characters
8 text = text.encode('ascii', 'ignore').decode('ascii')
9 return text
10
11# Handle various unicode issues
12texts = [
13 "café résumé", # Accented characters
14 "①②③", # Special numbers
15 "first floor", # Ligatures
16 ""smart quotes"" # Smart quotes
17]
18
19for t in texts:
20 print(f"{t} → {normalize_unicode(t)}")4. Contraction Expansion:
1contractions = {
2 "won't": "will not",
3 "can't": "cannot",
4 "n't": " not",
5 "'re": " are",
6 "'s": " is",
7 "'d": " would",
8 "'ll": " will",
9 "'ve": " have",
10 "'m": " am"
11}
12
13def expand_contractions(text: str) -> str:
14 """Expand contractions to full forms"""
15 for contraction, expansion in contractions.items():
16 text = text.replace(contraction, expansion)
17 return text
18
19text = "I won't say I can't do it. We're going and he's coming."
20print(f"Expanded: {expand_contractions(text)}")Best Practices:
Stemming and lemmatization reduce words to their base forms, decreasing vocabulary size and helping models understand word relationships.
Stemming:
Removes word suffixes using rule-based algorithms. Fast but can produce non-words.
1from nltk.stem import PorterStemmer, SnowballStemmer
2
3# Porter Stemmer (most common)
4porter = PorterStemmer()
5
6words = ['running', 'runs', 'ran', 'runner', 'easily', 'fairly',
7 'studies', 'studying', 'connection', 'connected']
8
9print("Porter Stemmer:")
10for word in words:
11 print(f" {word} → {porter.stem(word)}")
12
13# Snowball Stemmer (improved Porter)
14snowball = SnowballStemmer('english')
15
16print("\nSnowball Stemmer:")
17for word in words:
18 print(f" {word} → {snowball.stem(word)}")Stemming Algorithms Comparison:
| Algorithm | Speed | Accuracy | Example |
|-----------|-------|----------|---------|
| Porter | Fast | Moderate | "studies" → "studi" |
| Snowball | Fast | Better | "studies" → "studi" |
| Lancaster | Very Fast | Aggressive | "studies" → "stud" |
Lemmatization:
Reduces words to their dictionary form (lemma) using vocabulary and morphological analysis. More accurate but slower.
1from nltk.stem import WordNetLemmatizer
2from nltk.corpus import wordnet
3import nltk
4
5nltk.download('wordnet', quiet=True)
6nltk.download('averaged_perceptron_tagger', quiet=True)
7
8lemmatizer = WordNetLemmatizer()
9
10def get_wordnet_pos(tag):
11 """Convert NLTK POS tag to WordNet POS tag"""
12 tag_dict = {
13 'J': wordnet.ADJ,
14 'V': wordnet.VERB,
15 'N': wordnet.NOUN,
16 'R': wordnet.ADV
17 }
18 return tag_dict.get(tag[0], wordnet.NOUN)
19
20def lemmatize_with_pos(text: str) -> str:
21 """Lemmatize text with part-of-speech tagging"""
22 words = nltk.word_tokenize(text)
23 pos_tags = nltk.pos_tag(words)
24
25 lemmatized = []
26 for word, tag in pos_tags:
27 pos = get_wordnet_pos(tag)
28 lemmatized.append(lemmatizer.lemmatize(word, pos))
29
30 return ' '.join(lemmatized)
31
32# Compare stemming vs lemmatization
33test_words = ['better', 'running', 'studies', 'wolves', 'geese', 'went']
34
35print("Stemming vs Lemmatization:")
36print("-" * 50)
37for word in test_words:
38 stem = porter.stem(word)
39 # Lemmatize as verb for action words
40 pos = wordnet.VERB if word in ['running', 'studies', 'went'] else wordnet.NOUN
41 lemma = lemmatizer.lemmatize(word, pos)
42 print(f"{word:12} → Stem: {stem:10} | Lemma: {lemma}")spaCy Lemmatization (Recommended):
1import spacy
2
3nlp = spacy.load('en_core_web_sm')
4
5text = "The wolves were running quickly through the forest."
6doc = nlp(text)
7
8print("spaCy Lemmatization:")
9for token in doc:
10 if not token.is_punct and not token.is_space:
11 print(f" {token.text:12} → {token.lemma_:12} (POS: {token.pos_})")When to Use Each:
Use Stemming When:
Use Lemmatization When:
Key Differences:
| Aspect | Stemming | Lemmatization |
|--------|----------|---------------|
| Output | May not be real word | Always real word |
| Speed | Very fast | Slower |
| Accuracy | Lower | Higher |
| Requirements | Rules only | Dictionary + POS |
| Example | "better" → "better" | "better" → "good" |
Stopwords are common words that often don't carry significant meaning (e.g., "the", "is", "at"). Removing them reduces noise and computational cost.
Standard Stopword Removal:
1from nltk.corpus import stopwords
2import nltk
3
4nltk.download('stopwords', quiet=True)
5
6# NLTK stopwords
7english_stopwords = set(stopwords.words('english'))
8print(f"NLTK has {len(english_stopwords)} English stopwords")
9print(f"Sample: {list(english_stopwords)[:10]}")
10
11def remove_stopwords(tokens: list, custom_stopwords: set = None) -> list:
12 """Remove stopwords from token list"""
13 stop_words = english_stopwords.copy()
14 if custom_stopwords:
15 stop_words.update(custom_stopwords)
16
17 return [t for t in tokens if t.lower() not in stop_words]
18
19# Example
20tokens = ['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
21filtered = remove_stopwords(tokens)
22print(f"Original: {tokens}")
23print(f"Filtered: {filtered}")Custom Stopword Lists:
1def create_domain_stopwords(domain: str) -> set:
2 """Create domain-specific stopword lists"""
3 base = set(stopwords.words('english'))
4
5 domain_stopwords = {
6 'medical': {'patient', 'doctor', 'hospital', 'treatment'},
7 'legal': {'court', 'law', 'case', 'plaintiff', 'defendant'},
8 'tech': {'software', 'computer', 'system', 'data', 'user'},
9 'news': {'said', 'according', 'reported', 'statement'}
10 }
11
12 return base.union(domain_stopwords.get(domain, set()))
13
14# Domain-specific filtering
15tech_stopwords = create_domain_stopwords('tech')
16text = "The user said the software system was great for data processing"
17tokens = text.lower().split()
18filtered = [t for t in tokens if t not in tech_stopwords]
19print(f"Tech-filtered: {filtered}")When NOT to Remove Stopwords:
Some NLP tasks require stopwords:
Handling Rare Words:
1from collections import Counter
2
3def handle_rare_words(tokens: list, min_freq: int = 2,
4 unk_token: str = '<UNK>') -> tuple:
5 """Replace rare words with unknown token"""
6 # Count word frequencies
7 word_counts = Counter(tokens)
8
9 # Build vocabulary of frequent words
10 vocabulary = {word for word, count in word_counts.items()
11 if count >= min_freq}
12
13 # Replace rare words
14 processed = [t if t in vocabulary else unk_token for t in tokens]
15
16 return processed, vocabulary
17
18# Example with document collection
19documents = [
20 "the cat sat on the mat",
21 "the dog ran in the park",
22 "the cat and dog played together",
23 "a rare xenophobic comment appeared" # 'xenophobic' is rare
24]
25
26# Tokenize all documents
27all_tokens = []
28for doc in documents:
29 all_tokens.extend(doc.lower().split())
30
31processed, vocab = handle_rare_words(all_tokens, min_freq=2)
32print(f"Vocabulary size: {len(vocab)}")
33print(f"Vocabulary: {vocab}")
34print(f"Sample processed: {processed[-10:]}")Vocabulary Strategies:
1def build_vocabulary(documents: list,
2 max_vocab_size: int = 10000,
3 min_freq: int = 2) -> dict:
4 """Build vocabulary with size limits"""
5 # Count all words
6 word_counts = Counter()
7 for doc in documents:
8 tokens = doc.lower().split()
9 word_counts.update(tokens)
10
11 # Filter by frequency
12 frequent_words = {word: count for word, count in word_counts.items()
13 if count >= min_freq}
14
15 # Sort by frequency and take top N
16 sorted_words = sorted(frequent_words.items(),
17 key=lambda x: x[1], reverse=True)
18 top_words = sorted_words[:max_vocab_size - 2] # Reserve space for special tokens
19
20 # Create vocabulary mapping
21 vocab = {'<PAD>': 0, '<UNK>': 1}
22 for idx, (word, _) in enumerate(top_words, start=2):
23 vocab[word] = idx
24
25 return vocab
26
27# Build vocabulary
28vocab = build_vocabulary(documents, max_vocab_size=20)
29print(f"Vocabulary: {vocab}")Best Practices:
Production NLP systems require robust, reusable preprocessing pipelines. Let's build a complete pipeline class.
Complete Preprocessing Pipeline:
1import re
2import string
3from typing import List, Optional, Callable
4from dataclasses import dataclass
5import nltk
6from nltk.tokenize import word_tokenize
7from nltk.corpus import stopwords
8from nltk.stem import WordNetLemmatizer
9
10nltk.download('punkt', quiet=True)
11nltk.download('stopwords', quiet=True)
12nltk.download('wordnet', quiet=True)
13
14@dataclass
15class PreprocessingConfig:
16 """Configuration for text preprocessing"""
17 lowercase: bool = True
18 remove_punctuation: bool = True
19 remove_numbers: bool = False
20 remove_stopwords: bool = True
21 lemmatize: bool = True
22 min_word_length: int = 2
23 custom_stopwords: Optional[set] = None
24
25class TextPreprocessor:
26 """Complete text preprocessing pipeline"""
27
28 def __init__(self, config: PreprocessingConfig = None):
29 self.config = config or PreprocessingConfig()
30 self.lemmatizer = WordNetLemmatizer()
31 self.stopwords = set(stopwords.words('english'))
32 if self.config.custom_stopwords:
33 self.stopwords.update(self.config.custom_stopwords)
34
35 def clean_text(self, text: str) -> str:
36 """Basic text cleaning"""
37 # Remove URLs
38 text = re.sub(r'https?://\S+|www\.\S+', '', text)
39 # Remove emails
40 text = re.sub(r'\S+@\S+', '', text)
41 # Remove HTML tags
42 text = re.sub(r'<[^>]+>', '', text)
43 # Remove extra whitespace
44 text = ' '.join(text.split())
45 return text
46
47 def normalize(self, text: str) -> str:
48 """Normalize text"""
49 if self.config.lowercase:
50 text = text.lower()
51
52 if self.config.remove_punctuation:
53 text = text.translate(str.maketrans('', '', string.punctuation))
54
55 if self.config.remove_numbers:
56 text = re.sub(r'\d+', '', text)
57
58 return text
59
60 def tokenize(self, text: str) -> List[str]:
61 """Tokenize text"""
62 return word_tokenize(text)
63
64 def remove_stopwords(self, tokens: List[str]) -> List[str]:
65 """Remove stopwords"""
66 if not self.config.remove_stopwords:
67 return tokens
68 return [t for t in tokens if t.lower() not in self.stopwords]
69
70 def lemmatize(self, tokens: List[str]) -> List[str]:
71 """Lemmatize tokens"""
72 if not self.config.lemmatize:
73 return tokens
74 return [self.lemmatizer.lemmatize(t) for t in tokens]
75
76 def filter_tokens(self, tokens: List[str]) -> List[str]:
77 """Filter tokens by length"""
78 return [t for t in tokens if len(t) >= self.config.min_word_length]
79
80 def preprocess(self, text: str) -> List[str]:
81 """Complete preprocessing pipeline"""
82 # Step 1: Clean
83 text = self.clean_text(text)
84
85 # Step 2: Normalize
86 text = self.normalize(text)
87
88 # Step 3: Tokenize
89 tokens = self.tokenize(text)
90
91 # Step 4: Remove stopwords
92 tokens = self.remove_stopwords(tokens)
93
94 # Step 5: Lemmatize
95 tokens = self.lemmatize(tokens)
96
97 # Step 6: Filter
98 tokens = self.filter_tokens(tokens)
99
100 return tokens
101
102 def preprocess_batch(self, texts: List[str]) -> List[List[str]]:
103 """Preprocess multiple texts"""
104 return [self.preprocess(text) for text in texts]
105
106# Example usage
107config = PreprocessingConfig(
108 lowercase=True,
109 remove_punctuation=True,
110 remove_stopwords=True,
111 lemmatize=True,
112 min_word_length=2
113)
114
115preprocessor = TextPreprocessor(config)
116
117# Test texts
118texts = [
119 "The quick brown foxes were jumping over the lazy dogs!",
120 "Natural Language Processing (NLP) is amazing. Check https://example.com",
121 "I can't believe how easy this is! It's really helpful."
122]
123
124print("Preprocessing Results:")
125print("=" * 60)
126for text in texts:
127 tokens = preprocessor.preprocess(text)
128 print(f"Original: {text}")
129 print(f"Processed: {tokens}")
130 print()Scikit-learn Integration:
1from sklearn.base import BaseEstimator, TransformerMixin
2
3class SklearnTextPreprocessor(BaseEstimator, TransformerMixin):
4 """Scikit-learn compatible preprocessor"""
5
6 def __init__(self, lowercase=True, remove_stopwords=True):
7 self.lowercase = lowercase
8 self.remove_stopwords = remove_stopwords
9 self.preprocessor = None
10
11 def fit(self, X, y=None):
12 config = PreprocessingConfig(
13 lowercase=self.lowercase,
14 remove_stopwords=self.remove_stopwords
15 )
16 self.preprocessor = TextPreprocessor(config)
17 return self
18
19 def transform(self, X):
20 return [' '.join(self.preprocessor.preprocess(text)) for text in X]
21
22# Use in sklearn pipeline
23from sklearn.pipeline import Pipeline
24from sklearn.feature_extraction.text import TfidfVectorizer
25
26nlp_pipeline = Pipeline([
27 ('preprocess', SklearnTextPreprocessor()),
28 ('tfidf', TfidfVectorizer(max_features=1000))
29])
30
31# Fit and transform
32X = ["Sample text for processing", "Another example text"]
33X_transformed = nlp_pipeline.fit_transform(X)
34print(f"Transformed shape: {X_transformed.shape}")Pipeline Best Practices:
Build a production-ready text preprocessing pipeline that handles real-world text data including cleaning, normalization, tokenization, and lemmatization.
1import re
2import string
3from collections import Counter
4from typing import List, Dict, Optional
5import nltk
6from nltk.tokenize import word_tokenize, sent_tokenize
7from nltk.corpus import stopwords
8from nltk.stem import PorterStemmer, WordNetLemmatizer
9
10# Download required NLTK data
11nltk.download('punkt', quiet=True)
12nltk.download('stopwords', quiet=True)
13nltk.download('wordnet', quiet=True)
14
15print("=" * 70)
16print("COMPLETE TEXT PREPROCESSING PIPELINE")
17print("=" * 70)
18
19# ============================================================================
20# PREPROCESSING FUNCTIONS
21# ============================================================================
22
23class TextPreprocessingPipeline:
24 """Production-ready text preprocessing pipeline"""
25
26 def __init__(self):
27 self.stemmer = PorterStemmer()
28 self.lemmatizer = WordNetLemmatizer()
29 self.stop_words = set(stopwords.words('english'))
30
31 # Contraction mappings
32 self.contractions = {
33 "won't": "will not", "can't": "cannot", "n't": " not",
34 "'re": " are", "'s": " is", "'d": " would",
35 "'ll": " will", "'ve": " have", "'m": " am"
36 }
37
38 def clean_html(self, text: str) -> str:
39 """Remove HTML tags"""
40 return re.sub(r'<[^>]+>', '', text)
41
42 def remove_urls(self, text: str) -> str:
43 """Remove URLs"""
44 return re.sub(r'https?://\S+|www\.\S+', '[URL]', text)
45
46 def remove_emails(self, text: str) -> str:
47 """Remove email addresses"""
48 return re.sub(r'\S+@\S+', '[EMAIL]', text)
49
50 def expand_contractions(self, text: str) -> str:
51 """Expand contractions"""
52 for contraction, expansion in self.contractions.items():
53 text = text.replace(contraction, expansion)
54 return text
55
56 def normalize_whitespace(self, text: str) -> str:
57 """Normalize whitespace"""
58 return ' '.join(text.split())
59
60 def remove_special_chars(self, text: str, keep_punctuation: bool = False) -> str:
61 """Remove special characters"""
62 if keep_punctuation:
63 # Keep letters, numbers, basic punctuation
64 return re.sub(r'[^a-zA-Z0-9\s.,!?;:\'-]', '', text)
65 return re.sub(r'[^a-zA-Z0-9\s]', '', text)
66
67 def tokenize_words(self, text: str) -> List[str]:
68 """Tokenize into words"""
69 return word_tokenize(text.lower())
70
71 def tokenize_sentences(self, text: str) -> List[str]:
72 """Tokenize into sentences"""
73 return sent_tokenize(text)
74
75 def remove_stopwords(self, tokens: List[str],
76 additional_stopwords: Optional[set] = None) -> List[str]:
77 """Remove stopwords"""
78 stop = self.stop_words.copy()
79 if additional_stopwords:
80 stop.update(additional_stopwords)
81 return [t for t in tokens if t.lower() not in stop]
82
83 def stem_tokens(self, tokens: List[str]) -> List[str]:
84 """Apply stemming"""
85 return [self.stemmer.stem(t) for t in tokens]
86
87 def lemmatize_tokens(self, tokens: List[str]) -> List[str]:
88 """Apply lemmatization"""
89 return [self.lemmatizer.lemmatize(t) for t in tokens]
90
91 def filter_by_length(self, tokens: List[str],
92 min_len: int = 2, max_len: int = 50) -> List[str]:
93 """Filter tokens by length"""
94 return [t for t in tokens if min_len <= len(t) <= max_len]
95
96 def preprocess(self, text: str,
97 use_lemma: bool = True,
98 remove_stops: bool = True) -> Dict:
99 """Complete preprocessing pipeline"""
100
101 # Track original
102 original = text
103
104 # Step 1: Clean
105 text = self.clean_html(text)
106 text = self.remove_urls(text)
107 text = self.remove_emails(text)
108 text = self.expand_contractions(text)
109 text = self.normalize_whitespace(text)
110 cleaned = text
111
112 # Step 2: Tokenize
113 tokens = self.tokenize_words(text)
114
115 # Step 3: Remove special characters from tokens
116 tokens = [re.sub(r'[^a-zA-Z]', '', t) for t in tokens]
117 tokens = [t for t in tokens if t] # Remove empty
118
119 # Step 4: Remove stopwords
120 if remove_stops:
121 tokens = self.remove_stopwords(tokens)
122
123 # Step 5: Lemmatize or stem
124 if use_lemma:
125 tokens = self.lemmatize_tokens(tokens)
126 else:
127 tokens = self.stem_tokens(tokens)
128
129 # Step 6: Filter by length
130 tokens = self.filter_by_length(tokens)
131
132 return {
133 'original': original,
134 'cleaned': cleaned,
135 'tokens': tokens,
136 'num_tokens': len(tokens)
137 }
138
139# ============================================================================
140# TEST THE PIPELINE
141# ============================================================================
142
143pipeline = TextPreprocessingPipeline()
144
145# Sample texts with various challenges
146test_texts = [
147 "The quick brown foxes were jumping over the lazy dogs! It's amazing.",
148 "Check out https://example.com for more info! Contact: [email protected]",
149 "<p>This is <b>HTML</b> content</p> with extra spaces.",
150 "I can't believe it won't work... We've tried everything! #frustrating",
151 "Natural Language Processing (NLP) is a subfield of AI. It's fascinating!"
152]
153
154print("\n1. PREPROCESSING RESULTS")
155print("=" * 70)
156
157for i, text in enumerate(test_texts, 1):
158 result = pipeline.preprocess(text)
159 print(f"\nExample {i}:")
160 print(f" Original: {result['original'][:60]}...")
161 print(f" Cleaned: {result['cleaned'][:60]}...")
162 print(f" Tokens: {result['tokens']}")
163 print(f" Count: {result['num_tokens']} tokens")
164
165# ============================================================================
166# COMPARE STEMMING VS LEMMATIZATION
167# ============================================================================
168
169print("\n\n2. STEMMING VS LEMMATIZATION COMPARISON")
170print("=" * 70)
171
172comparison_words = [
173 'running', 'runs', 'ran', 'runner',
174 'better', 'best', 'good',
175 'studies', 'studying', 'studied',
176 'wolves', 'children', 'geese'
177]
178
179print(f"{'Word':<15} {'Stem':<15} {'Lemma':<15}")
180print("-" * 45)
181for word in comparison_words:
182 stem = pipeline.stemmer.stem(word)
183 lemma = pipeline.lemmatizer.lemmatize(word)
184 print(f"{word:<15} {stem:<15} {lemma:<15}")
185
186# ============================================================================
187# VOCABULARY ANALYSIS
188# ============================================================================
189
190print("\n\n3. VOCABULARY ANALYSIS")
191print("=" * 70)
192
193# Combine all tokens
194all_tokens = []
195for text in test_texts:
196 result = pipeline.preprocess(text)
197 all_tokens.extend(result['tokens'])
198
199# Analyze vocabulary
200word_freq = Counter(all_tokens)
201vocabulary = set(all_tokens)
202
203print(f"Total tokens: {len(all_tokens)}")
204print(f"Unique words (vocabulary): {len(vocabulary)}")
205print(f"Vocabulary: {sorted(vocabulary)}")
206print(f"\nMost common words:")
207for word, count in word_freq.most_common(10):
208 print(f" {word}: {count}")
209
210# ============================================================================
211# BATCH PROCESSING EXAMPLE
212# ============================================================================
213
214print("\n\n4. BATCH PROCESSING")
215print("=" * 70)
216
217# Larger corpus
218corpus = [
219 "Machine learning is transforming industries worldwide.",
220 "Deep learning models require large amounts of data.",
221 "Natural language processing enables computers to understand text.",
222 "Computer vision allows machines to interpret images.",
223 "Reinforcement learning trains agents through rewards and penalties."
224]
225
226print(f"Processing {len(corpus)} documents...")
227
228# Process batch
229processed_corpus = []
230for doc in corpus:
231 result = pipeline.preprocess(doc)
232 processed_corpus.append(result['tokens'])
233
234# Analyze corpus statistics
235total_tokens = sum(len(doc) for doc in processed_corpus)
236all_vocab = set()
237for doc in processed_corpus:
238 all_vocab.update(doc)
239
240print(f"\nCorpus Statistics:")
241print(f" Documents: {len(corpus)}")
242print(f" Total tokens: {total_tokens}")
243print(f" Vocabulary size: {len(all_vocab)}")
244print(f" Avg tokens/doc: {total_tokens / len(corpus):.1f}")
245
246print("\nProcessed documents:")
247for i, (original, processed) in enumerate(zip(corpus, processed_corpus), 1):
248 print(f" {i}. {processed}")
249
250print("\n" + "=" * 70)
251print("PREPROCESSING PIPELINE COMPLETE")
252print("=" * 70)This comprehensive preprocessing pipeline demonstrates production-ready NLP text processing:
Pipeline Architecture:
1. Cleaning Stage:
2. Tokenization Stage:
3. Normalization Stage:
4. Stopword Removal:
5. Lemmatization/Stemming:
6. Filtering:
Key Design Decisions:
Configurability:
Traceability:
Performance:
Best Practices Demonstrated:
Excellent foundation in text preprocessing! You now understand tokenization, normalization, stemming, lemmatization, and building preprocessing pipelines. In our next lesson, "Word Embeddings: Word2Vec & GloVe," you'll learn how to represent words as dense vectors that capture semantic meaning, enabling models to understand word relationships and similarity.
Your progress is saved in this browser, on this device only.