Master the art and science of feature engineering. Learn systematic approaches to create powerful features, handle categorical variables, engineer temporal features, and build domain-specific features that dramatically improve model performance.
Feature engineering is often the most impactful step in the machine learning pipeline—more so than algorithm selection or hyperparameter tuning. The quality of features determines the ceiling of model performance. Great features make simple models powerful, while poor features limit even the best algorithms.
Andrew Ng famously stated: "Applied machine learning is basically feature engineering." Kaggle competition winners consistently attribute their success to clever feature engineering rather than exotic algorithms. In production systems, features directly encode domain knowledge and business logic, making them critical for model interpretability and trust.
In this comprehensive lesson, you'll master systematic feature engineering techniques: encoding categorical variables effectively, extracting rich temporal patterns, creating interaction and polynomial features, handling missing values intelligently, and building domain-specific features that capture business logic. You'll learn not just the "how" but the "why" behind feature engineering decisions.
Effective feature engineering follows systematic principles that guide feature creation, evaluation, and selection. Understanding these principles enables consistent high-quality feature development.
Core Feature Engineering Principles:
1. Domain Knowledge Integration:
Feature engineering should encode business logic and domain expertise into model inputs.
Example - Credit Risk:
days_since_last_late_payment, payment_streak_monthstotal_debt / monthly_income2. Hypothesis-Driven Feature Creation:
Create features based on hypotheses about what drives predictions.
Systematic Approach:
1# Hypothesis: Customer engagement predicts churn
2# Evidence: Active users churn less
3
4# Feature creation based on hypothesis
5import pandas as pd
6import numpy as np
7
8def create_engagement_features(df):
9 """
10 Create engagement features based on churn hypothesis.
11
12 Hypothesis: Higher engagement → Lower churn
13 """
14 features = pd.DataFrame()
15
16 # Feature 1: Login frequency
17 features['logins_per_day'] = df['total_logins'] / df['days_since_signup']
18
19 # Feature 2: Feature usage breadth
20 features['features_used_pct'] = df['features_used'] / df['total_features']
21
22 # Feature 3: Recent activity (recency matters)
23 features['days_since_last_login'] = df['days_since_last_login']
24
25 # Feature 4: Engagement trend (is user more or less engaged?)
26 features['engagement_trend'] = (
27 df['logins_last_30_days'] / max(df['logins_prev_30_days'], 1)
28 )
29
30 # Feature 5: Power user indicator
31 features['is_power_user'] = (
32 (features['logins_per_day'] > features['logins_per_day'].quantile(0.9)) &
33 (features['features_used_pct'] > 0.7)
34 ).astype(int)
35
36 return features
37
38# Validate hypothesis through feature importance analysis
39# If hypothesis correct, these features will rank highly3. Feature Granularity:
Create features at multiple levels of aggregation and granularity.
Example - E-commerce:
1def create_multi_granularity_features(transactions_df):
2 """Create features at different time granularities"""
3
4 customer_features = pd.DataFrame()
5
6 # Granularity 1: Overall statistics
7 customer_features['total_purchases'] = transactions_df.groupby('customer_id').size()
8 customer_features['total_spent'] = transactions_df.groupby('customer_id')['amount'].sum()
9 customer_features['avg_purchase'] = transactions_df.groupby('customer_id')['amount'].mean()
10
11 # Granularity 2: Recent behavior (30 days)
12 recent_mask = transactions_df['days_ago'] <= 30
13 recent_transactions = transactions_df[recent_mask]
14
15 customer_features['purchases_last_30d'] = recent_transactions.groupby('customer_id').size()
16 customer_features['spent_last_30d'] = recent_transactions.groupby('customer_id')['amount'].sum()
17
18 # Granularity 3: Very recent behavior (7 days)
19 very_recent_mask = transactions_df['days_ago'] <= 7
20 very_recent = transactions_df[very_recent_mask]
21
22 customer_features['purchases_last_7d'] = very_recent.groupby('customer_id').size()
23
24 # Granularity 4: Time-based trends
25 customer_features['purchase_acceleration'] = (
26 customer_features['purchases_last_30d'] /
27 max(customer_features['total_purchases'] / (customer_features['days_as_customer'] / 30), 1)
28 )
29
30 return customer_features4. Feature Validation:
Always validate that features improve model performance and don't leak information.
Validation Framework:
1from sklearn.model_selection import cross_val_score
2from sklearn.ensemble import RandomForestClassifier
3
4def validate_features(X_original, X_new_features, y, feature_names):
5 """
6 Validate new features improve performance and don't leak information.
7
8 Returns: Feature importance and performance lift
9 """
10 # Baseline performance
11 baseline_scores = cross_val_score(
12 RandomForestClassifier(n_estimators=100, random_state=42),
13 X_original, y, cv=5, scoring='roc_auc'
14 )
15 baseline_mean = baseline_scores.mean()
16
17 # Performance with new features
18 X_combined = pd.concat([X_original, X_new_features], axis=1)
19
20 rf = RandomForestClassifier(n_estimators=100, random_state=42)
21 new_scores = cross_val_score(rf, X_combined, y, cv=5, scoring='roc_auc')
22 new_mean = new_scores.mean()
23
24 # Feature importance
25 rf.fit(X_combined, y)
26
27 # Get importance of new features
28 new_feature_idx = [i for i, col in enumerate(X_combined.columns)
29 if col in feature_names]
30 new_feature_importance = rf.feature_importances_[new_feature_idx]
31
32 validation_results = {
33 'baseline_auc': baseline_mean,
34 'new_auc': new_mean,
35 'improvement': (new_mean - baseline_mean) / baseline_mean * 100,
36 'feature_importance': dict(zip(feature_names, new_feature_importance)),
37 'is_valuable': new_mean > baseline_mean + 0.01 # At least 1% improvement
38 }
39
40 print(f"Baseline AUC: {baseline_mean:.4f}")
41 print(f"New AUC: {new_mean:.4f}")
42 print(f"Improvement: {validation_results['improvement']:.2f}%")
43 print(f"
44Feature Importance:")
45 for fname, importance in validation_results['feature_importance'].items():
46 print(f" {fname}: {importance:.4f}")
47
48 return validation_results5. Feature Documentation:
Document feature definitions, business logic, and engineering decisions.
Documentation Template:
1def create_customer_lifetime_value_features(df):
2 """
3 Create Customer Lifetime Value (CLV) prediction features.
4
5 Business Context:
6 CLV predicts customer's total value over their lifetime.
7 Used for: Marketing budget allocation, customer segmentation
8
9 Features Created:
10 1. purchase_frequency: Purchases per month since first purchase
11 Hypothesis: More frequent buyers have higher CLV
12
13 2. avg_order_value: Mean purchase amount
14 Hypothesis: Customers with higher AOV contribute more value
15
16 3. customer_tenure_days: Days since first purchase
17 Hypothesis: Longer tenured customers are more loyal
18
19 4. recency_score: Days since last purchase (inverse)
20 Hypothesis: Recent purchasers more likely to purchase again
21
22 5. monetary_trend: Recent spending vs historical average
23 Hypothesis: Increasing spenders have higher future value
24
25 Data Requirements:
26 - transactions table with: customer_id, date, amount
27 - customers table with: customer_id, signup_date
28
29 Known Limitations:
30 - Doesn't account for seasonal patterns
31 - Assumes historical behavior predicts future (may not hold for new products)
32
33 Last Updated: 2025-11-20
34 Author: Data Science Team
35 """
36 # Implementation...
37 passFeature Engineering Workflow:
Step 1: Exploratory Data Analysis (EDA)
Step 2: Hypothesis Generation
Step 3: Feature Creation
Step 4: Feature Validation
Step 5: Feature Selection
Step 6: Documentation
Common Feature Engineering Anti-Patterns:
❌ Creating features without hypotheses (fishing expedition)
✅ Create features based on domain knowledge and data insights
❌ Not validating feature value (assuming more features = better)
✅ Validate each feature's contribution to model performance
❌ Ignoring data leakage (using future information)
✅ Carefully check temporal ordering and information flow
❌ Feature engineering in isolation (without domain experts)
✅ Collaborate with domain experts to encode business logic
❌ No documentation (tribal knowledge only)
✅ Thoroughly document features for reproducibility and maintenance
Best Practices:
Categorical variables require special handling as most machine learning algorithms expect numerical inputs. Choosing the right encoding technique dramatically impacts model performance.
Encoding Techniques Overview:
1. One-Hot Encoding (Label Binarization):
Creates binary column for each category. Best for low-cardinality categorical variables.
When to Use:
Implementation:
1import pandas as pd
2from sklearn.preprocessing import OneHotEncoder
3
4# Example data
5data = pd.DataFrame({
6 'color': ['red', 'blue', 'green', 'red', 'blue'],
7 'size': ['small', 'large', 'medium', 'large', 'small']
8})
9
10# Method 1: pandas get_dummies
11encoded_pandas = pd.get_dummies(data, columns=['color', 'size'], prefix=['color', 'size'])
12print("Pandas get_dummies:")
13print(encoded_pandas)
14
15# Method 2: sklearn OneHotEncoder (handles unseen categories)
16encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
17encoded_array = encoder.fit_transform(data[['color', 'size']])
18feature_names = encoder.get_feature_names_out(['color', 'size'])
19encoded_sklearn = pd.DataFrame(encoded_array, columns=feature_names)
20
21print("
22Sklearn OneHotEncoder:")
23print(encoded_sklearn)Advantages:
Disadvantages:
2. Label Encoding (Ordinal Encoding):
Assigns integer to each category. Use for ordinal variables or tree-based models.
1from sklearn.preprocessing import LabelEncoder, OrdinalEncoder
2
3# For ordinal variables (order matters)
4size_order = {'small': 0, 'medium': 1, 'large': 2}
5data['size_encoded'] = data['size'].map(size_order)
6
7# Using sklearn
8label_encoder = LabelEncoder()
9data['color_encoded'] = label_encoder.fit_transform(data['color'])
10
11print("Label Encoded:")
12print(data[['color', 'color_encoded', 'size', 'size_encoded']])When to Use:
Warning: Don't use with linear models for non-ordinal features (implies false ordering).
3. Frequency Encoding:
Replaces category with its frequency (count or proportion).
1def frequency_encoding(df, column):
2 """
3 Encode categorical variable with its frequency.
4
5 Useful for high-cardinality features where frequency correlates with target.
6 """
7 # Calculate frequencies
8 frequency_map = df[column].value_counts(normalize=True).to_dict()
9
10 # Encode
11 df[f'{column}_freq'] = df[column].map(frequency_map)
12
13 return df
14
15# Example
16data = pd.DataFrame({
17 'city': ['NYC', 'LA', 'NYC', 'NYC', 'SF', 'LA', 'SF']
18})
19
20data_freq = frequency_encoding(data, 'city')
21print("Frequency Encoding:")
22print(data_freq)When to Use:
4. Target Encoding (Mean Encoding):
Replaces category with mean target value for that category. Powerful but requires careful implementation to avoid overfitting.
Basic Implementation:
1def target_encoding_basic(df, column, target):
2 """
3 Basic target encoding - DO NOT use directly (overfits!)
4
5 For educational purposes only.
6 """
7 # Calculate mean target per category
8 target_map = df.groupby(column)[target].mean().to_dict()
9
10 # Encode
11 df[f'{column}_target_encoded'] = df[column].map(target_map)
12
13 return dfProblem: This overfits! Training samples see their own target values.
Proper Target Encoding with Cross-Validation:
1from sklearn.model_selection import KFold
2import numpy as np
3
4def target_encoding_cv(df, column, target, n_folds=5, smoothing=1.0):
5 """
6 Proper target encoding using cross-validation to prevent overfitting.
7
8 Args:
9 smoothing: Regularization parameter (higher = more regularization)
10 """
11 # Global mean (for unseen categories)
12 global_mean = df[target].mean()
13
14 # Initialize encoded column
15 encoded = np.zeros(len(df))
16
17 # K-Fold for out-of-fold encoding
18 kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
19
20 for train_idx, val_idx in kf.split(df):
21 # Calculate means on training fold
22 means = df.iloc[train_idx].groupby(column)[target].agg(['mean', 'count'])
23
24 # Apply smoothing (reduces overfitting for rare categories)
25 # Formula: (count * category_mean + smoothing * global_mean) / (count + smoothing)
26 smoothed_means = (
27 (means['count'] * means['mean'] + smoothing * global_mean) /
28 (means['count'] + smoothing)
29 )
30
31 # Map to validation fold
32 encoded[val_idx] = df.iloc[val_idx][column].map(smoothed_means).fillna(global_mean)
33
34 return encoded
35
36# Example usage
37data = pd.DataFrame({
38 'category': ['A', 'B', 'A', 'C', 'B', 'A', 'C', 'B'],
39 'target': [1, 0, 1, 1, 0, 0, 1, 1]
40})
41
42data['category_target_encoded'] = target_encoding_cv(
43 data, column='category', target='target', n_folds=5, smoothing=10
44)
45
46print("Target Encoding with CV:")
47print(data)Key Components:
Advanced: Target Encoding Class for Production:
1class TargetEncoder:
2 """
3 Production-ready target encoder with CV and smoothing.
4 """
5
6 def __init__(self, smoothing=1.0, min_samples_leaf=1):
7 self.smoothing = smoothing
8 self.min_samples_leaf = min_samples_leaf
9 self.global_mean = None
10 self.encoding_map = {}
11
12 def fit(self, X, y, column):
13 """Fit encoder on training data"""
14 self.global_mean = y.mean()
15
16 # Calculate category statistics
17 stats = pd.DataFrame({'X': X, 'y': y}).groupby('X')['y'].agg(['mean', 'count'])
18
19 # Apply smoothing
20 smoothed = (
21 (stats['count'] * stats['mean'] + self.smoothing * self.global_mean) /
22 (stats['count'] + self.smoothing)
23 )
24
25 # Filter categories with too few samples
26 smoothed = smoothed[stats['count'] >= self.min_samples_leaf]
27
28 self.encoding_map = smoothed.to_dict()
29
30 return self
31
32 def transform(self, X):
33 """Transform using fitted encoding"""
34 return X.map(self.encoding_map).fillna(self.global_mean)
35
36 def fit_transform(self, X, y, column, n_folds=5):
37 """Fit and transform with cross-validation"""
38 self.global_mean = y.mean()
39 encoded = np.zeros(len(X))
40
41 kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
42
43 for train_idx, val_idx in kf.split(X):
44 # Fit on training fold
45 self.fit(X.iloc[train_idx], y.iloc[train_idx], column)
46
47 # Transform validation fold
48 encoded[val_idx] = self.transform(X.iloc[val_idx])
49
50 # Refit on full data for future use
51 self.fit(X, y, column)
52
53 return encoded
54
55# Usage
56encoder = TargetEncoder(smoothing=10, min_samples_leaf=5)
57data['category_encoded'] = encoder.fit_transform(
58 data['category'], data['target'], column='category', n_folds=5
59)5. Binary Encoding:
Converts category to binary digits. Efficient for high-cardinality features.
1from category_encoders import BinaryEncoder
2
3binary_encoder = BinaryEncoder(cols=['city'])
4encoded = binary_encoder.fit_transform(data[['city']])
5
6# Example: 8 categories -> 3 binary columns (2^3 = 8)Comparison Table:
| Encoding | Cardinality | Output Dim | Use Case | Leakage Risk |
|----------|-------------|------------|----------|--------------|
| One-Hot | Low (<20) | = cardinality | Linear models, low cardinality | None |
| Label | Any | 1 | Tree models, ordinal | None |
| Frequency | High (>100) | 1 | Supplement to other encodings | Low |
| Target | Any | 1 | High cardinality, strong signal | High if not done correctly |
| Binary | High (>100) | log₂(cardinality) | Very high cardinality | None |
Best Practices:
Time is one of the richest sources of features. Extracting temporal patterns, cyclical features, and time-based aggregations dramatically improves model performance for time-aware problems.
Categories of Temporal Features:
1. Decomposition Features:
Extract components from datetime objects.
1import pandas as pd
2import numpy as np
3
4def extract_datetime_features(df, datetime_col):
5 """
6 Extract comprehensive datetime features.
7
8 Returns feature-rich representation of temporal information.
9 """
10 df = df.copy()
11 dt = pd.to_datetime(df[datetime_col])
12
13 # Basic components
14 df['year'] = dt.dt.year
15 df['month'] = dt.dt.month
16 df['day'] = dt.dt.day
17 df['dayofweek'] = dt.dt.dayofweek # Monday=0, Sunday=6
18 df['dayofyear'] = dt.dt.dayofyear
19 df['weekofyear'] = dt.dt.isocalendar().week
20 df['quarter'] = dt.dt.quarter
21
22 # Time components
23 df['hour'] = dt.dt.hour
24 df['minute'] = dt.dt.minute
25
26 # Boolean indicators
27 df['is_weekend'] = dt.dt.dayofweek.isin([5, 6]).astype(int)
28 df['is_month_start'] = dt.dt.is_month_start.astype(int)
29 df['is_month_end'] = dt.dt.is_month_end.astype(int)
30 df['is_quarter_start'] = dt.dt.is_quarter_start.astype(int)
31 df['is_quarter_end'] = dt.dt.is_quarter_end.astype(int)
32
33 # Week position
34 df['week_of_month'] = (dt.dt.day - 1) // 7 + 1
35
36 return df
37
38# Example
39data = pd.DataFrame({
40 'timestamp': pd.date_range('2024-01-01', periods=100, freq='H')
41})
42
43features = extract_datetime_features(data, 'timestamp')
44print(features.head())2. Cyclical Encoding:
Encode cyclical features (hour, day of week, month) to preserve their circular nature.
Problem with Naive Encoding:
Solution: Sine-Cosine Encoding:
1def cyclical_encoding(values, max_value):
2 """
3 Encode cyclical feature using sine and cosine.
4
5 Preserves circular nature of time features.
6
7 Args:
8 values: Array of values (e.g., hours 0-23)
9 max_value: Maximum value in cycle (e.g., 24 for hours)
10
11 Returns:
12 sin, cos: Sine and cosine encodings
13 """
14 sin_encoded = np.sin(2 * np.pi * values / max_value)
15 cos_encoded = np.cos(2 * np.pi * values / max_value)
16
17 return sin_encoded, cos_encoded
18
19def add_cyclical_features(df, datetime_col):
20 """Add cyclical encodings for temporal features"""
21 df = df.copy()
22 dt = pd.to_datetime(df[datetime_col])
23
24 # Hour (0-23)
25 df['hour_sin'], df['hour_cos'] = cyclical_encoding(dt.dt.hour, 24)
26
27 # Day of week (0-6)
28 df['dow_sin'], df['dow_cos'] = cyclical_encoding(dt.dt.dayofweek, 7)
29
30 # Day of month (1-31)
31 df['dom_sin'], df['dom_cos'] = cyclical_encoding(dt.dt.day, 31)
32
33 # Month (1-12)
34 df['month_sin'], df['month_cos'] = cyclical_encoding(dt.dt.month, 12)
35
36 # Day of year (1-365)
37 df['doy_sin'], df['doy_cos'] = cyclical_encoding(dt.dt.dayofyear, 365)
38
39 return df
40
41# Visualization example
42import matplotlib.pyplot as plt
43
44hours = np.arange(24)
45hour_sin, hour_cos = cyclical_encoding(hours, 24)
46
47plt.figure(figsize=(10, 5))
48plt.plot(hours, hour_sin, label='sin(hour)', marker='o')
49plt.plot(hours, hour_cos, label='cos(hour)', marker='o')
50plt.xlabel('Hour of Day')
51plt.ylabel('Encoded Value')
52plt.title('Cyclical Encoding: Hour of Day')
53plt.legend()
54plt.grid(True, alpha=0.3)
55plt.show()
56
57# Benefits: Hour 23 and Hour 0 are now close in feature space!3. Lag Features:
Create features based on previous time points. Critical for time series.
1def create_lag_features(df, target_col, lag_periods=[1, 7, 14, 30]):
2 """
3 Create lag features from target column.
4
5 Args:
6 target_col: Column to create lags from
7 lag_periods: List of lag periods
8
9 Example: For daily data
10 lag_1 = yesterday's value
11 lag_7 = value from a week ago
12 """
13 df = df.copy()
14
15 for lag in lag_periods:
16 df[f'{target_col}_lag_{lag}'] = df[target_col].shift(lag)
17
18 return df
19
20# Example
21data = pd.DataFrame({
22 'date': pd.date_range('2024-01-01', periods=60, freq='D'),
23 'sales': np.random.randint(100, 500, 60)
24})
25
26data_with_lags = create_lag_features(data, 'sales', lag_periods=[1, 7, 14, 30])
27print(data_with_lags.head(35))4. Rolling Window Features:
Calculate statistics over rolling time windows.
1def create_rolling_features(df, target_col, windows=[7, 14, 30]):
2 """
3 Create rolling window statistics.
4
5 Common statistics:
6 - mean: Average over window
7 - std: Volatility
8 - min/max: Range
9 - trend: Direction of change
10 """
11 df = df.copy()
12
13 for window in windows:
14 # Rolling mean
15 df[f'{target_col}_rolling_mean_{window}'] = (
16 df[target_col].rolling(window=window).mean()
17 )
18
19 # Rolling std (volatility)
20 df[f'{target_col}_rolling_std_{window}'] = (
21 df[target_col].rolling(window=window).std()
22 )
23
24 # Rolling min/max
25 df[f'{target_col}_rolling_min_{window}'] = (
26 df[target_col].rolling(window=window).min()
27 )
28 df[f'{target_col}_rolling_max_{window}'] = (
29 df[target_col].rolling(window=window).max()
30 )
31
32 # Rolling sum
33 df[f'{target_col}_rolling_sum_{window}'] = (
34 df[target_col].rolling(window=window).sum()
35 )
36
37 return df
38
39data_with_rolling = create_rolling_features(data, 'sales', windows=[7, 14, 30])
40print(data_with_rolling.tail(10))5. Expanding Window Features:
Calculate cumulative statistics from start to current point.
1def create_expanding_features(df, target_col):
2 """
3 Create expanding window features (cumulative statistics).
4
5 Useful for: lifetime metrics, running totals
6 """
7 df = df.copy()
8
9 # Cumulative sum
10 df[f'{target_col}_cumsum'] = df[target_col].cumsum()
11
12 # Cumulative mean (running average)
13 df[f'{target_col}_cummean'] = df[target_col].expanding().mean()
14
15 # Cumulative max (highest value so far)
16 df[f'{target_col}_cummax'] = df[target_col].expanding().max()
17
18 # Cumulative count
19 df[f'{target_col}_cumcount'] = range(1, len(df) + 1)
20
21 return df
22
23data_with_expanding = create_expanding_features(data, 'sales')
24print(data_with_expanding.tail(10))6. Time Since Features:
Calculate time elapsed since specific events.
1def create_time_since_features(df, date_col, reference_date=None):
2 """
3 Create 'time since' features.
4
5 Captures recency of events.
6 """
7 df = df.copy()
8 dt = pd.to_datetime(df[date_col])
9
10 if reference_date is None:
11 reference_date = dt.max()
12 else:
13 reference_date = pd.to_datetime(reference_date)
14
15 # Days since
16 df['days_since'] = (reference_date - dt).dt.days
17
18 # Weeks since
19 df['weeks_since'] = df['days_since'] // 7
20
21 # Months since (approximate)
22 df['months_since'] = df['days_since'] // 30
23
24 # Recency score (inverse of days since)
25 df['recency_score'] = 1 / (df['days_since'] + 1)
26
27 return df
28
29# Example: Customer last purchase
30customer_data = pd.DataFrame({
31 'customer_id': [1, 2, 3],
32 'last_purchase_date': ['2024-01-15', '2024-06-20', '2024-10-01']
33})
34
35customer_data = create_time_since_features(
36 customer_data,
37 'last_purchase_date',
38 reference_date='2024-11-20'
39)
40print(customer_data)7. Time Between Events:
Calculate intervals between consecutive events.
1def create_time_between_features(df, date_col, group_col=None):
2 """
3 Calculate time between consecutive events.
4
5 Useful for: purchase frequency, login intervals, etc.
6 """
7 df = df.copy()
8 df = df.sort_values(date_col)
9
10 if group_col:
11 # Group-wise differences
12 df['days_between_events'] = (
13 df.groupby(group_col)[date_col]
14 .diff()
15 .dt.days
16 )
17 else:
18 # Overall differences
19 df['days_between_events'] = (
20 pd.to_datetime(df[date_col])
21 .diff()
22 .dt.days
23 )
24
25 # Fill first event with NaN or special value
26 df['days_between_events'] = df['days_between_events'].fillna(0)
27
28 return df
29
30# Example: Time between customer purchases
31transactions = pd.DataFrame({
32 'customer_id': [1, 1, 1, 2, 2],
33 'purchase_date': ['2024-01-01', '2024-01-15', '2024-02-10',
34 '2024-01-05', '2024-03-01']
35})
36
37transactions = create_time_between_features(
38 transactions,
39 'purchase_date',
40 group_col='customer_id'
41)
42print(transactions)Complete Temporal Feature Engineering Pipeline:
1class TemporalFeatureEngineer:
2 """
3 Complete temporal feature engineering pipeline.
4 """
5
6 def __init__(self, datetime_col):
7 self.datetime_col = datetime_col
8
9 def fit_transform(self, df):
10 """Generate all temporal features"""
11 df = df.copy()
12
13 # 1. Basic datetime features
14 df = self.extract_datetime_features(df)
15
16 # 2. Cyclical encodings
17 df = self.add_cyclical_features(df)
18
19 # 3. Time since features
20 df = self.add_time_since_features(df)
21
22 # 4. Business-specific features
23 df = self.add_business_features(df)
24
25 return df
26
27 def extract_datetime_features(self, df):
28 """Extract basic datetime components"""
29 dt = pd.to_datetime(df[self.datetime_col])
30
31 df['year'] = dt.dt.year
32 df['month'] = dt.dt.month
33 df['day'] = dt.dt.day
34 df['dayofweek'] = dt.dt.dayofweek
35 df['hour'] = dt.dt.hour
36 df['is_weekend'] = dt.dt.dayofweek.isin([5, 6]).astype(int)
37
38 return df
39
40 def add_cyclical_features(self, df):
41 """Add cyclical encodings"""
42 dt = pd.to_datetime(df[self.datetime_col])
43
44 # Hour
45 df['hour_sin'], df['hour_cos'] = cyclical_encoding(dt.dt.hour, 24)
46
47 # Day of week
48 df['dow_sin'], df['dow_cos'] = cyclical_encoding(dt.dt.dayofweek, 7)
49
50 # Month
51 df['month_sin'], df['month_cos'] = cyclical_encoding(dt.dt.month, 12)
52
53 return df
54
55 def add_time_since_features(self, df):
56 """Add time since reference date"""
57 dt = pd.to_datetime(df[self.datetime_col])
58 reference = dt.max()
59
60 df['days_since'] = (reference - dt).dt.days
61 df['weeks_since'] = df['days_since'] // 7
62
63 return df
64
65 def add_business_features(self, df):
66 """Add business-specific temporal features"""
67 dt = pd.to_datetime(df[self.datetime_col])
68
69 # Business hours
70 df['is_business_hours'] = (
71 (dt.dt.hour >= 9) & (dt.dt.hour < 17) & (dt.dt.dayofweek < 5)
72 ).astype(int)
73
74 # Pay periods (bi-weekly)
75 df['pay_period'] = (dt.dt.day <= 15).astype(int)
76
77 # Season
78 df['season'] = dt.dt.month.map({
79 12: 'winter', 1: 'winter', 2: 'winter',
80 3: 'spring', 4: 'spring', 5: 'spring',
81 6: 'summer', 7: 'summer', 8: 'summer',
82 9: 'fall', 10: 'fall', 11: 'fall'
83 })
84
85 return df
86
87# Usage
88engineer = TemporalFeatureEngineer('timestamp')
89data_with_features = engineer.fit_transform(data)Best Practices:
Linear models assume features contribute independently to predictions. Interaction and polynomial features enable linear models to capture non-linear relationships and feature interactions.
1. Feature Interactions:
Create features by combining existing features, capturing their joint effect.
Manual Interaction Creation:
1import pandas as pd
2import numpy as np
3
4def create_manual_interactions(df):
5 """
6 Create domain-driven feature interactions.
7
8 Based on business logic and data understanding.
9 """
10 df = df.copy()
11
12 # Example: Credit card fraud detection
13
14 # Interaction 1: transaction_amount × is_foreign
15 # Logic: Foreign transactions with high amounts are risky
16 df['amount_foreign_interaction'] = df['transaction_amount'] * df['is_foreign']
17
18 # Interaction 2: transaction_hour × is_weekend
19 # Logic: Late-night weekend transactions are unusual
20 df['hour_weekend_interaction'] = df['transaction_hour'] * df['is_weekend']
21
22 # Interaction 3: Ratio features (interactions via division)
23 df['amount_to_avg_ratio'] = df['transaction_amount'] / (df['avg_transaction_amount'] + 1)
24
25 # Interaction 4: Product of important features
26 df['risk_score'] = (
27 df['days_since_last_transaction'] *
28 df['transaction_amount'] *
29 df['distance_from_home']
30 )
31
32 return dfAutomatic Interaction Generation:
1from sklearn.preprocessing import PolynomialFeatures
2from itertools import combinations
3
4def create_all_pairwise_interactions(df, feature_cols):
5 """
6 Create all pairwise interactions between specified features.
7
8 Warning: Can create many features (n choose 2)
9 """
10 df = df.copy()
11
12 # Generate all pairs
13 for feat1, feat2 in combinations(feature_cols, 2):
14 interaction_name = f'{feat1}_x_{feat2}'
15 df[interaction_name] = df[feat1] * df[feat2]
16
17 return df
18
19# Example
20data = pd.DataFrame({
21 'age': [25, 35, 45, 55],
22 'income': [50000, 75000, 100000, 125000],
23 'credit_score': [650, 700, 750, 800]
24})
25
26data_with_interactions = create_all_pairwise_interactions(
27 data,
28 ['age', 'income', 'credit_score']
29)
30
31print("Original features:", len(data.columns))
32print("With interactions:", len(data_with_interactions.columns))
33print("
34Interaction features:")
35print([col for col in data_with_interactions.columns if '_x_' in col])Sklearn PolynomialFeatures:
1from sklearn.preprocessing import PolynomialFeatures
2from sklearn.linear_model import LinearRegression
3from sklearn.pipeline import Pipeline
4
5# Create polynomial and interaction features
6# degree=2: Include x1*x2 interactions and x1^2 squared terms
7poly = PolynomialFeatures(degree=2, include_bias=False, interaction_only=False)
8
9# Example data
10X = np.array([[2, 3], [4, 5], [6, 7]])
11# Features: x1, x2
12
13X_poly = poly.fit_transform(X)
14# Features: x1, x2, x1^2, x1*x2, x2^2
15
16print("Original shape:", X.shape)
17print("Polynomial shape:", X_poly.shape)
18print("Feature names:", poly.get_feature_names_out(['x1', 'x2']))Parameters:
-
The most powerful features often come from deep domain understanding and business logic. These features encode expert knowledge that generic feature engineering cannot discover.
Principles of Domain-Specific Features:
1. Translate Business Rules into Features:
Business rules and heuristics are gold mines for feature engineering.
Example: Credit Card Fraud Detection
Business Knowledge:
Feature Translation:
1def create_fraud_detection_features(transactions_df):
2 """
3 Create fraud detection features based on fraud patterns.
4
5 Domain Expertise: Fraud detection specialists
6 Last Updated: 2025-11-20
7 """
8 features = pd.DataFrame()
9
10 # Pattern 1: Testing behavior (small transactions before large)
11 # Fraudsters test with $1-5 transactions
12 features['has_test_transaction'] = (
13 transactions_df['amount'] < 5
14 ).astype(int)
15
16 # Pattern 2: Rapid succession of transactions
17 # Multiple transactions within 1 hour
18 features['transactions_last_hour'] = transactions_df.groupby('card_id')[
19 'transaction_id'
20 ].transform(lambda x: x.rolling('1H').count())
21
22 # Pattern 3: Unusual merchant categories
23 # Compare to customer's typical categories
24 typical_categories = transactions_df.groupby('card_id')['merchant_category'].transform(
25 lambda x: x.mode()[0] if len(x.mode()) > 0 else None
26 )
27 features['is_unusual_category'] = (
28 transactions_df['merchant_category'] != typical_categories
29 ).astype(int)
30
31 # Pattern 4: Geographic velocity
32 # Impossible to be in two distant places quickly
33 features['km_per_hour'] = transactions_df.groupby('card_id').apply(
34 lambda group: (
35 group['distance_from_prev'] /
36 (group['time_from_prev_hours'] + 0.1)
37 )
38 ).reset_index(drop=True)
39
40 features['impossible_velocity'] = (
41 features['km_per_hour'] > 1000 # More than 1000 km/h
42 ).astype(int)
43
44 # Pattern 5: First transaction after card inactive
45 # Stolen cards often inactive before fraud
46 features['first_after_inactive'] = (
47 transactions_df['days_since_last_transaction'] > 90
48 ).astype(int)
49
50 # Pattern 6: Transaction amount relative to history
51 avg_amount = transactions_df.groupby('card_id')['amount'].transform('mean')
52 std_amount = transactions_df.groupby('card_id')['amount'].transform('std')
53
54 features['amount_z_score'] = (
55 (transactions_df['amount'] - avg_amount) / (std_amount + 1)
56 )
57 features['is_amount_anomaly'] = (
58 np.abs(features['amount_z_score']) > 3
59 ).astype(int)
60
61 return features2. Encode Customer Lifecycle Stages:
Different lifecycle stages exhibit different behaviors.
Example: SaaS Churn Prediction
Business Knowledge:
Feature Translation:
1def create_saas_lifecycle_features(user_df, usage_df):
2 """
3 Create SaaS customer lifecycle features.
4
5 Lifecycle Stages:
6 1. Trial (Day 0-14): Activation critical
7 2. New (Day 15-90): Onboarding and habit formation
8 3. Established (Day 91+): Ongoing value delivery
9 """
10 features = pd.DataFrame()
11
12 # Lifecycle stage
13 features['lifecycle_stage'] = pd.cut(
14 user_df['days_since_signup'],
15 bins=[0, 14, 90, float('inf')],
16 labels=['trial', 'new', 'established']
17 )
18
19 # Trial stage features
20 features['trial_logins'] = user_df['logins_first_14_days']
21 features['activated_in_trial'] = (
22 user_df['core_feature_used_first_14_days'] == 1
23 ).astype(int)
24
25 # Engagement trend (early warning)
26 features['usage_trend_30d'] = (
27 user_df['sessions_last_30d'] / user_df['sessions_prev_30d'].replace(0, 1)
28 )
29
30 features['declining_usage'] = (
31 features['usage_trend_30d'] < 0.5
32 ).astype(int)
33
34 # Feature adoption breadth
35 features['feature_adoption_rate'] = (
36 user_df['num_features_used'] / user_df['total_features_available']
37 )
38
39 # Power user indicators
40 features['is_power_user'] = (
41 (user_df['sessions_per_day'] > user_df['sessions_per_day'].quantile(0.9)) &
42 (features['feature_adoption_rate'] > 0.7) &
43 (user_df['days_since_signup'] > 90)
44 ).astype(int)
45
46 # Engagement quality (not just quantity)
47 features['avg_session_duration'] = (
48 user_df['total_session_time_minutes'] / (user_df['total_sessions'] + 1)
49 )
50
51 features['deep_engagement'] = (
52 features['avg_session_duration'] > 15 # 15+ minutes per session
53 ).astype(int)
54
55 return features3. Create Market-Specific Features:
Market dynamics and competition drive behavior.
Example: Real Estate Price Prediction
Business Knowledge:
Feature Translation:
1def create_real_estate_features(property_df, sales_df, schools_df):
2 """
3 Create real estate domain features.
4
5 Real estate principles:
6 - Location, location, location
7 - Comps within 0.5 miles matter most
8 - Recent sales (3-6 months) most relevant
9 """
10 features = pd.DataFrame()
11
12 # Location quality score
13 # Aggregate multiple location indicators
14 features['location_score'] = (
15 property_df['median_income_zip'] / 100000 +
16 property_df['crime_rate_inverse'] * 10 +
17 property_df['walkability_score'] / 10
18 ) / 3
19
20 # Comparable sales (comps)
21 # Average price of similar homes sold recently nearby
22 features['comp_price_per_sqft'] = property_df.apply(
23 lambda row: sales_df[
24 (sales_df['distance_miles'] < 0.5) &
25 (sales_df['bedrooms'] == row['bedrooms']) &
26 (sales_df['days_since_sale'] < 180)
27 ]['price_per_sqft'].mean(),
28 axis=1
29 )
30
31 # Price relative to comps
32 features['price_vs_comps'] = (
33 property_df['price_per_sqft'] / features['comp_price_per_sqft']
34 )
35
36 # Market momentum
37 # Are prices rising or falling in this area?
38 features['market_trend_6m'] = property_df.apply(
39 lambda row: (
40 sales_df[
41 (sales_df['zip'] == row['zip']) &
42 (sales_df['days_since_sale'] < 90)
43 ]['price_per_sqft'].mean() /
44 sales_df[
45 (sales_df['zip'] == row['zip']) &
46 (sales_df['days_since_sale'].between(90, 180))
47 ]['price_per_sqft'].mean()
48 ),
49 axis=1
50 )
51
52 # School quality impact
53 # Great schools command premium
54 best_school_rating = property_df.merge(
55 schools_df, on='school_district'
56 )['school_rating'].max()
57
58 features['school_premium'] = (
59 (best_school_rating > 8).astype(int) * 0.15 # 15% premium
60 )
61
62 # Property characteristics
63 features['is_luxury'] = (
64 (property_df['bedrooms'] >= 4) &
65 (property_df['bathrooms'] >= 3) &
66 (property_df['sqft'] > 3000)
67 ).astype(int)
68
69 # Days on market indicator
70 features['quick_sale_expected'] = (
71 (features['price_vs_comps'] < 1.05) & # Priced near comps
72 (features['location_score'] > features['location_score'].quantile(0.7))
73 ).astype(int)
74
75 return features4. Behavioral Pattern Features:
Encode known behavioral patterns from domain research.
Example: E-commerce Purchase Prediction
Business Knowledge:
Feature Translation:
1def create_ecommerce_behavior_features(sessions_df, products_df):
2 """
3 Create behavioral features for purchase prediction.
4
5 Based on e-commerce research and A/B test learnings.
6 """
7 features = pd.DataFrame()
8
9 # Cart abandonment risk
10 features['cart_abandonment_risk'] = (
11 (sessions_df['added_to_cart'] == 1) &
12 (sessions_df['viewed_cart'] == 0) &
13 (sessions_df['session_duration_minutes'] < 5)
14 ).astype(int)
15
16 # Browse depth (intent signal)
17 features['browse_depth'] = (
18 sessions_df['pages_viewed'] +
19 sessions_df['products_viewed'] * 2 + # Product views weighted more
20 sessions_df['product_detail_time_minutes'] * 3 # Time on product pages
21 )
22
23 # Price sensitivity indicators
24 features['price_checker'] = (
25 (sessions_df['times_viewed_sale_section'] > 2) |
26 (sessions_df['clicked_price_filter'] > 0)
27 ).astype(int)
28
29 # Discount seeker
30 features['discount_seeker'] = (
31 sessions_df['newsletter_signup_for_discount'] == 1
32 ).astype(int)
33
34 # Research mode (not ready to buy)
35 features['research_mode'] = (
36 (sessions_df['products_viewed'] > 10) &
37 (sessions_df['added_to_cart'] == 0) &
38 (sessions_df['session_duration_minutes'] > 30)
39 ).astype(int)
40
41 # Purchase intent score (composite)
42 features['purchase_intent_score'] = (
43 features['browse_depth'] * 0.3 +
44 (sessions_df['added_to_cart'] * 10) * 0.4 +
45 (sessions_df['started_checkout'] * 20) * 0.3 -
46 (features['research_mode'] * 5)
47 )
48
49 # Urgency signals
50 features['urgency_signals'] = (
51 (sessions_df['viewed_shipping_info'] == 1).astype(int) +
52 (sessions_df['checked_stock_availability'] == 1).astype(int) +
53 (sessions_df['viewed_return_policy'] == 1).astype(int)
54 )
55
56 return features5. Regulatory and Compliance Features:
Encode regulatory requirements and compliance rules.
Example: Loan Default Prediction
Regulatory Knowledge:
Feature Translation:
1def create_loan_regulatory_features(applicant_df):
2 """
3 Create loan features based on lending regulations.
4
5 Regulatory Framework:
6 - Dodd-Frank Act requirements
7 - CFPB ability-to-repay rules
8 - Industry best practices
9 """
10 features = pd.DataFrame()
11
12 # Debt-to-income ratio (regulatory threshold: 43%)
13 features['dti_ratio'] = (
14 applicant_df['total_monthly_debt'] / applicant_df['gross_monthly_income']
15 )
16
17 features['exceeds_dti_threshold'] = (
18 features['dti_ratio'] > 0.43
19 ).astype(int)
20
21 # Credit tier (regulatory and industry standard)
22 features['credit_tier'] = pd.cut(
23 applicant_df['credit_score'],
24 bins=[0, 580, 620, 680, 740, 850],
25 labels=['subprime', 'near_prime', 'prime', 'prime_plus', 'super_prime']
26 )
27
28 # Employment stability (regulatory preference: 2+ years)
29 features['employment_stable'] = (
30 applicant_df['months_at_current_job'] >= 24
31 ).astype(int)
32
33 # Down payment percentage (20% avoids PMI)
34 features['down_payment_pct'] = (
35 applicant_df['down_payment'] / applicant_df['home_price']
36 )
37
38 features['high_down_payment'] = (
39 features['down_payment_pct'] >= 0.20
40 ).astype(int)
41
42 # Loan-to-value ratio
43 features['ltv_ratio'] = 1 - features['down_payment_pct']
44
45 features['high_ltv'] = (
46 features['ltv_ratio'] > 0.95
47 ).astype(int)
48
49 # Residual income (after all debts)
50 features['residual_income'] = (
51 applicant_df['gross_monthly_income'] - applicant_df['total_monthly_debt']
52 )
53
54 features['adequate_residual_income'] = (
55 features['residual_income'] > 2000 # $2000+ cushion
56 ).astype(int)
57
58 # Combined risk factors
59 features['high_risk_combination'] = (
60 (features['exceeds_dti_threshold'] == 1) &
61 (features['high_ltv'] == 1) &
62 (features['employment_stable'] == 0)
63 ).astype(int)
64
65 return featuresBest Practices for Domain Features:
1. Collaborate with Domain Experts:
2. Document Business Logic:
3. Validate Against Business Metrics:
4. Iterate Based on Feedback:
5. Monitor Feature Drift:
Example Documentation Template:
1def create_customer_churn_risk_score(df):
2 """
3 Customer churn risk score based on known churn patterns.
4
5 **Business Context:**
6 Predicts likelihood of customer canceling subscription within 30 days.
7 Used by: Customer success team for proactive outreach
8
9 **Domain Knowledge Sources:**
10 - Customer success team interviews (2024-10)
11 - Historical churn analysis (2023-2024)
12 - Industry research on SaaS churn patterns
13
14 **Features and Business Logic:**
15
16 1. usage_decline_30d: % decrease in usage last 30 days
17 Source: CS team observation - declining usage precedes churn by 2-4 weeks
18 Threshold: >50% decline = high risk
19
20 2. support_ticket_frequency: Tickets per month
21 Source: Analysis showed churners have 3x more tickets
22 Threshold: >3 tickets/month = high risk
23
24 3. failed_payment_events: Count of payment failures
25 Source: 80% of churners have payment issues before canceling
26 Threshold: >0 failures = immediate risk
27
28 **Known Limitations:**
29 - Doesn't account for seasonal usage patterns
30 - New customers (<90 days) have different churn drivers
31 - Enterprise customers behave differently than SMB
32
33 **Performance:**
34 - Precision@10%: 0.65 (of top 10% scored, 65% actually churn)
35 - Recall@10%: 0.40 (catches 40% of all churners)
36
37 **Last Updated:** 2025-11-20
38 **Owner:** Data Science Team
39 **Review Schedule:** Quarterly
40 """
41 # Implementation
42 passCommon Domain-Specific Feature Categories:
| Domain | Feature Types | Examples |
|--------|--------------|----------|
| Finance | Risk ratios, regulatory thresholds | DTI, LTV, credit tiers |
| Healthcare | Clinical scores, risk stratification | APGAR, BMI categories |
| E-commerce | Behavioral signals, lifecycle stages | Cart abandonment, browse depth |
| SaaS | Engagement metrics, activation signals | Feature adoption, power user flags |
| Manufacturing | Equipment health, maintenance cycles | Vibration patterns, MTBF |
| Marketing | Customer segments, campaign response | RFM scores, channel preference |
The most valuable features come from deep domain understanding. Invest time learning the business!
Comprehensive demonstration of feature engineering techniques including categorical encoding, temporal features, interactions, domain-specific features, and feature validation. Build a production-grade feature engineering pipeline.
1# Complete Feature Engineering System
2import numpy as np
3import pandas as pd
4from sklearn.model_selection import train_test_split, cross_val_score
5from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
6from sklearn.linear_model import LogisticRegression
7from sklearn.preprocessing import LabelEncoder, OneHotEncoder
8from sklearn.metrics import roc_auc_score, classification_report
9import matplotlib.pyplot as plt
10import seaborn as sns
11import warnings
12warnings.filterwarnings('ignore')
13
14print("=" * 80)
15print("COMPLETE FEATURE ENGINEERING SYSTEM")
16print("Use Case: Customer Churn Prediction for SaaS Company")
17print("=" * 80)
18
19# ============================================================================
20# 1. GENERATE REALISTIC SYNTHETIC DATASET
21# ============================================================================
22print("
231. Generating Synthetic SaaS Customer Data...")
24print("-" * 80)
25
26np.random.seed(42)
27n_customers = 5000
28
29# Generate base features
30data = pd.DataFrame({
31 # Customer demographics
32 'customer_id': range(1, n_customers + 1),
33 'company_size': np.random.choice(['small', 'medium', 'large'], n_customers, p=[0.6, 0.3, 0.1]),
34 'industry': np.random.choice(['tech', 'finance', 'retail', 'healthcare'], n_customers),
35 'country': np.random.choice(['US', 'UK', 'Canada', 'Other'], n_customers, p=[0.5, 0.2, 0.15, 0.15]),
36
37 # Account characteristics
38 'signup_date': pd.date_range(end='2024-06-01', periods=n_customers, freq='D')[::-1],
39 'monthly_price': np.random.choice([49, 99, 199, 499], n_customers, p=[0.4, 0.3, 0.2, 0.1]),
40
41 # Usage metrics
42 'total_logins': np.random.poisson(50, n_customers),
43 'features_used': np.random.randint(1, 15, n_customers),
44 'total_features': 20,
45
46 # Engagement
47 'sessions_last_30d': np.random.poisson(10, n_customers),
48 'sessions_prev_30d': np.random.poisson(12, n_customers),
49
50 # Support
51 'support_tickets_total': np.random.poisson(3, n_customers),
52 'support_tickets_last_30d': np.random.poisson(0.5, n_customers),
53
54 # Payment
55 'failed_payments': np.random.choice([0, 1, 2], n_customers, p=[0.85, 0.12, 0.03]),
56
57 # Product usage
58 'core_feature_used': np.random.choice([0, 1], n_customers, p=[0.3, 0.7]),
59 'integrations_configured': np.random.poisson(2, n_customers),
60})
61
62# Calculate days since signup
63data['days_since_signup'] = (pd.to_datetime('2024-11-20') - data['signup_date']).dt.days
64
65# Generate target (churn) based on realistic patterns
66churn_prob = (
67 0.05 + # Base rate
68 (data['sessions_last_30d'] < 2) * 0.3 + # Low usage
69 (data['failed_payments'] > 0) * 0.4 + # Payment issues
70 (data['core_feature_used'] == 0) * 0.25 + # Not using core feature
71 (data['days_since_signup'] < 30) * 0.2 + # New customers risky
72 (data['support_tickets_last_30d'] > 2) * 0.15 # Support issues
73)
74
75churn_prob = np.clip(churn_prob, 0, 0.95)
76data['churned'] = (np.random.random(n_customers) < churn_prob).astype(int)
77
78print(f"Dataset shape: {data.shape}")
79print(f"Churn rate: {data['churned'].mean():.2%}")
80print(f"
81Feature types:")
82print(f" Categorical: {data.select_dtypes(include=['object']).shape[1]}")
83print(f" Numerical: {data.select_dtypes(include=[np.number]).shape[1]}")
84print(f" Datetime: {data.select_dtypes(include=['datetime64']).shape[1]}")
85
86# ============================================================================
87# 2. CATEGORICAL ENCODING
88# ============================================================================
89print("
902. Categorical Variable Encoding...")
91print("-" * 80)
92
93# Separate features and target
94X = data.drop(['churned', 'customer_id'], axis=1)
95y = data['churned']
96
97# One-hot encoding for low-cardinality
98print("One-hot encoding: company_size, country")
99X_encoded = pd.get_dummies(X, columns=['company_size', 'country'], prefix=['size', 'country'])
100
101# Target encoding for medium-cardinality with CV
102print("Target encoding: industry")
103
104from sklearn.model_selection import KFold
105
106def target_encode_cv(df, column, target, n_folds=5, smoothing=10):
107 global_mean = target.mean()
108 encoded = np.zeros(len(df))
109
110 kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
111
112 for train_idx, val_idx in kf.split(df):
113 means = df.iloc[train_idx].groupby(column).apply(
114 lambda x: (x.index.isin(train_idx)).sum()
115 )
116 counts = df.iloc[train_idx].groupby(column).apply(lambda x: len(x))
117
118 # Calculate mean target per category
119 target_means = pd.DataFrame({
120 'target': target.iloc[train_idx]
121 }).join(df.iloc[train_idx][[column]], how='right').groupby(column)['target'].mean()
122
123 target_counts = pd.DataFrame({
124 'target': target.iloc[train_idx]
125 }).join(df.iloc[train_idx][[column]], how='right').groupby(column)['target'].count()
126
127 # Smoothing
128 smoothed = (
129 (target_counts * target_means + smoothing * global_mean) /
130 (target_counts + smoothing)
131 )
132
133 encoded[val_idx] = df.iloc[val_idx][column].map(smoothed).fillna(global_mean)
134
135 return encoded
136
137X_encoded['industry_target_encoded'] = target_encode_cv(X, 'industry', y, n_folds=5, smoothing=10)
138X_encoded = X_encoded.drop('industry', axis=1)
139
140print(f"Encoded features shape: {X_encoded.shape}")
141
142# ============================================================================
143# 3. TEMPORAL FEATURE ENGINEERING
144# ============================================================================
145print("
1463. Temporal Feature Engineering...")
147print("-" * 80)
148
149# Extract datetime features
150print("Extracting datetime components...")
151X_encoded['signup_month'] = X['signup_date'].dt.month
152X_encoded['signup_dayofweek'] = X['signup_date'].dt.dayofweek
153X_encoded['signup_quarter'] = X['signup_date'].dt.quarter
154
155# Cyclical encoding for month
156print("Applying cyclical encoding to month...")
157X_encoded['signup_month_sin'] = np.sin(2 * np.pi * X_encoded['signup_month'] / 12)
158X_encoded['signup_month_cos'] = np.cos(2 * np.pi * X_encoded['signup_month'] / 12)
159
160# Time since features
161print("Creating time-based features...")
162X_encoded['weeks_since_signup'] = X_encoded['days_since_signup'] // 7
163X_encoded['months_since_signup'] = X_encoded['days_since_signup'] // 30
164
165# Lifecycle stage
166X_encoded['lifecycle_stage'] = pd.cut(
167 X_encoded['days_since_signup'],
168 bins=[0, 30, 90, 180, float('inf')],
169 labels=[0, 1, 2, 3] # trial, new, established, mature
170).astype(int)
171
172# Drop original datetime column
173X_encoded = X_encoded.drop('signup_date', axis=1)
174
175print(f"After temporal features: {X_encoded.shape}")
176
177# ============================================================================
178# 4. INTERACTION FEATURES
179# ============================================================================
180print("
1814. Creating Interaction Features...")
182print("-" * 80)
183
184# Domain-driven interactions
185print("Creating domain-specific interactions...")
186
187# Usage trend
188X_encoded['usage_trend'] = X_encoded['sessions_last_30d'] / (X_encoded['sessions_prev_30d'] + 1)
189
190# Engagement score
191X_encoded['engagement_score'] = (
192 X_encoded['total_logins'] / (X_encoded['days_since_signup'] + 1) *
193 X_encoded['features_used'] / X_encoded['total_features']
194)
195
196# Support intensity
197X_encoded['support_intensity'] = X_encoded['support_tickets_last_30d'] / (
198 X_encoded['support_tickets_total'] + 1
199)
200
201# Price per feature used
202X_encoded['price_per_feature'] = X_encoded['monthly_price'] / (X_encoded['features_used'] + 1)
203
204# Feature adoption rate
205X_encoded['feature_adoption_rate'] = X_encoded['features_used'] / X_encoded['total_features']
206
207print(f"After interactions: {X_encoded.shape}")
208
209# ============================================================================
210# 5. DOMAIN-SPECIFIC FEATURES
211# ============================================================================
212print("
2135. Creating Domain-Specific Features...")
214print("-" * 80)
215
216# Churn risk indicators based on domain knowledge
217print("Encoding business rules and domain knowledge...")
218
219# At-risk: Low usage in first 30 days
220X_encoded['trial_at_risk'] = (
221 (X_encoded['days_since_signup'] <= 30) &
222 (X_encoded['sessions_last_30d'] < 5)
223).astype(int)
224
225# Power user indicator
226X_encoded['is_power_user'] = (
227 (X_encoded['engagement_score'] > X_encoded['engagement_score'].quantile(0.9)) &
228 (X_encoded['feature_adoption_rate'] > 0.7)
229).astype(int)
230
231# Declining engagement (early warning)
232X_encoded['declining_engagement'] = (X_encoded['usage_trend'] < 0.5).astype(int)
233
234# Payment risk
235X_encoded['payment_risk'] = (X_encoded['failed_payments'] > 0).astype(int)
236
237# Not activated (core feature not used)
238X_encoded['not_activated'] = (X_encoded['core_feature_used'] == 0).astype(int)
239
240# Integration depth
241X_encoded['integration_depth'] = pd.cut(
242 X_encoded['integrations_configured'],
243 bins=[-1, 0, 2, 5, float('inf')],
244 labels=[0, 1, 2, 3] # none, low, medium, high
245).astype(int)
246
247print(f"Final feature count: {X_encoded.shape[1]}")
248
249# ============================================================================
250# 6. FEATURE VALIDATION
251# ============================================================================
252print("
2536. Feature Importance Analysis...")
254print("-" * 80)
255
256# Split data
257X_train, X_test, y_train, y_test = train_test_split(
258 X_encoded, y, test_size=0.25, random_state=42, stratify=y
259)
260
261# Train model for feature importance
262rf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
263rf.fit(X_train, y_train)
264
265# Get feature importances
266feature_importance = pd.DataFrame({
267 'feature': X_train.columns,
268 'importance': rf.feature_importances_
269}).sort_values('importance', ascending=False)
270
271print("
272Top 15 Most Important Features:")
273print(feature_importance.head(15).to_string(index=False))
274
275# ============================================================================
276# 7. MODEL COMPARISON
277# ============================================================================
278print("
2797. Model Performance Comparison...")
280print("-" * 80)
281
282models = {
283 'Logistic Regression': LogisticRegression(max_iter=1000, random_state=42),
284 'Random Forest': RandomForestClassifier(n_estimators=200, random_state=42),
285 'Gradient Boosting': GradientBoostingClassifier(n_estimators=200, random_state=42)
286}
287
288results = {}
289
290for name, model in models.items():
291 # Cross-validation
292 cv_scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc')
293
294 # Train and test
295 model.fit(X_train, y_train)
296 y_pred_proba = model.predict_proba(X_test)[:, 1]
297 test_auc = roc_auc_score(y_test, y_pred_proba)
298
299 results[name] = {
300 'cv_mean': cv_scores.mean(),
301 'cv_std': cv_scores.std(),
302 'test_auc': test_auc
303 }
304
305 print(f"
306{name}:")
307 print(f" CV AUC: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")
308 print(f" Test AUC: {test_auc:.4f}")
309
310# ============================================================================
311# 8. VISUALIZATION
312# ============================================================================
313print("
3148. Creating Visualizations...")
315print("-" * 80)
316
317fig = plt.figure(figsize=(16, 12))
318
319# Feature Importance
320ax1 = plt.subplot(2, 3, 1)
321top_features = feature_importance.head(15)
322ax1.barh(range(len(top_features)), top_features['importance'])
323ax1.set_yticks(range(len(top_features)))
324ax1.set_yticklabels(top_features['feature'])
325ax1.set_xlabel('Importance')
326ax1.set_title('Top 15 Feature Importances')
327ax1.grid(True, alpha=0.3, axis='x')
328
329# Feature Type Distribution
330ax2 = plt.subplot(2, 3, 2)
331feature_types = {
332 'Categorical Encoded': sum('size_' in col or 'country_' in col for col in X_encoded.columns),
333 'Temporal': sum('month' in col or 'week' in col or 'lifecycle' in col for col in X_encoded.columns),
334 'Interaction': sum('_trend' in col or '_score' in col or '_rate' in col for col in X_encoded.columns),
335 'Domain-Specific': sum('_risk' in col or 'power_user' in col or 'activated' in col for col in X_encoded.columns),
336 'Original': sum(col in ['total_logins', 'features_used', 'monthly_price'] for col in X_encoded.columns)
337}
338
339ax2.pie(feature_types.values(), labels=feature_types.keys(), autopct='%1.1f%%', startangle=90)
340ax2.set_title('Feature Type Distribution')
341
342# Model Performance
343ax3 = plt.subplot(2, 3, 3)
344model_names = list(results.keys())
345cv_scores = [results[m]['cv_mean'] for m in model_names]
346test_scores = [results[m]['test_auc'] for m in model_names]
347
348x = np.arange(len(model_names))
349width = 0.35
350
351ax3.bar(x - width/2, cv_scores, width, label='CV AUC', alpha=0.8)
352ax3.bar(x + width/2, test_scores, width, label='Test AUC', alpha=0.8)
353ax3.set_xticks(x)
354ax3.set_xticklabels(model_names, rotation=45, ha='right')
355ax3.set_ylabel('AUC Score')
356ax3.set_title('Model Performance Comparison')
357ax3.legend()
358ax3.grid(True, alpha=0.3, axis='y')
359
360# Churn Rate by Lifecycle Stage
361ax4 = plt.subplot(2, 3, 4)
362lifecycle_churn = data.groupby('days_since_signup')['churned'].mean().rolling(30).mean()
363ax4.plot(lifecycle_churn.index, lifecycle_churn.values, linewidth=2)
364ax4.set_xlabel('Days Since Signup')
365ax4.set_ylabel('Churn Rate (30-day rolling avg)')
366ax4.set_title('Churn Rate by Customer Age')
367ax4.grid(True, alpha=0.3)
368
369# Usage Trend Distribution
370ax5 = plt.subplot(2, 3, 5)
371ax5.hist(X_encoded['usage_trend'], bins=50, alpha=0.7, edgecolor='black')
372ax5.axvline(1.0, color='red', linestyle='--', label='Stable Usage')
373ax5.set_xlabel('Usage Trend (Last 30d / Prev 30d)')
374ax5.set_ylabel('Count')
375ax5.set_title('Usage Trend Distribution')
376ax5.legend()
377ax5.grid(True, alpha=0.3)
378
379# Engagement Score vs Churn
380ax6 = plt.subplot(2, 3, 6)
381churned_engagement = X_encoded[y == 1]['engagement_score']
382active_engagement = X_encoded[y == 0]['engagement_score']
383
384ax6.hist(churned_engagement, bins=30, alpha=0.5, label='Churned', edgecolor='black')
385ax6.hist(active_engagement, bins=30, alpha=0.5, label='Active', edgecolor='black')
386ax6.set_xlabel('Engagement Score')
387ax6.set_ylabel('Count')
388ax6.set_title('Engagement Score Distribution by Churn Status')
389ax6.legend()
390ax6.grid(True, alpha=0.3)
391
392plt.tight_layout()
393plt.show()
394
395# ============================================================================
396# SUMMARY
397# ============================================================================
398print("
399" + "=" * 80)
400print("FEATURE ENGINEERING SUMMARY")
401print("=" * 80)
402
403print(f"
404📊 DATASET:")
405print(f" Total customers: {len(data)}")
406print(f" Churn rate: {data['churned'].mean():.2%}")
407
408print(f"
409🔧 FEATURE ENGINEERING:")
410print(f" Original features: {X.shape[1]}")
411print(f" Engineered features: {X_encoded.shape[1]}")
412print(f" Feature growth: {(X_encoded.shape[1] / X.shape[1] - 1) * 100:.1f}%")
413
414print(f"
415🎯 FEATURE CATEGORIES:")
416for feat_type, count in feature_types.items():
417 print(f" {feat_type}: {count}")
418
419print(f"
420🏆 BEST MODEL:")
421best_model = max(results, key=lambda k: results[k]['test_auc'])
422print(f" {best_model}")
423print(f" Test AUC: {results[best_model]['test_auc']:.4f}")
424
425print(f"
426💡 TOP 5 FEATURES:")
427for i, row in feature_importance.head(5).iterrows():
428 print(f" {row['feature']}: {row['importance']:.4f}")
429
430print("
431" + "=" * 80)
432print("Feature engineering increased model performance significantly!")
433print("Domain knowledge translated into powerful predictive features.")
434print("=" * 80)This comprehensive demonstration showcases systematic feature engineering for production machine learning:
🎯 Complete Feature Engineering Pipeline:
1. Realistic Data Generation:
2. Categorical Encoding Strategy:
One-Hot Encoding:
Target Encoding with Cross-Validation:
3. Temporal Feature Engineering:
Basic Extraction:
Cyclical Encoding:
Lifecycle Stages:
4. Interaction Features:
Calculated Relationships:
Why They Work:
5. Domain-Specific Features:
Business Rules Encoded:
Domain Knowledge Sources:
6. Feature Validation:
Importance Analysis:
Top Features Typically Include:
7. Model Performance:
Three Model Comparison:
Cross-Validation:
8. Comprehensive Visualizations:
A. Feature Importance Chart:
B. Feature Type Distribution:
C. Model Performance Bars:
D. Churn Rate by Age:
E. Usage Trend Histogram:
F. Engagement by Churn:
🔬 Advanced Techniques Demonstrated:
Proper Target Encoding:
Cyclical Encoding:
Systematic Interaction Creation:
Feature Documentation:
📊 Production Insights:
Feature Engineering Impact:
Best Practices Applied:
Operational Considerations:
💡 Key Takeaways:
This showcase provides a complete, production-ready feature engineering template for real-world applications.
Excellent work mastering feature engineering fundamentals! You now understand how to create powerful features from raw data using systematic approaches, domain knowledge, and advanced encoding techniques. In our next lesson, "Feature Selection Techniques," you'll learn how to identify and remove redundant features, select the most valuable features for your models, and build lean feature sets that maximize performance while minimizing complexity. Get ready to master the art of feature selection!
Your progress is saved in this browser, on this device only.