Master advanced bagging techniques, random forest optimization, and ensemble diversity strategies for superior predictive performance.
Bagging (Bootstrap Aggregating) and Random Forests represent the foundation of modern ensemble learning. While basic random forests are easy to implement, achieving state-of-the-art performance requires deep understanding of ensemble theory, hyperparameter optimization, and advanced techniques for controlling model diversity and preventing overfitting.
Random Forests, invented by Leo Breiman in 2001, remain one of the most powerful and widely-used machine learning algorithms in production systems. They excel at both regression and classification, handle high-dimensional data naturally, provide built-in feature importance, and are remarkably resistant to overfitting. Major tech companies rely on random forests for everything from fraud detection to recommendation systems.
In this comprehensive lesson, you'll move beyond basic random forest usage to master advanced optimization techniques, understand the mathematical foundations of bagging, implement sophisticated feature selection strategies, and build production-grade ensemble systems that consistently outperform single models.
Bagging reduces prediction variance by training multiple models on bootstrap samples and averaging their predictions. Understanding the theory enables optimal ensemble design.
Bootstrap Sampling Fundamentals:
Bootstrap sampling creates training sets by randomly sampling with replacement from the original dataset. Each bootstrap sample is the same size as the original data, but contains approximately 63.2% unique samples (the rest are duplicates).
Mathematical Foundation:
For a dataset with n samples, the probability a given sample is NOT selected in one draw is (1 - 1/n). After n draws with replacement:
1P(sample not selected) = (1 - 1/n)^n → 1/e ≈ 0.368 as n → ∞
2P(sample selected) ≈ 0.632 or 63.2%Variance Reduction Mechanism:
Given m independent models with variance σ² and correlation ρ, the ensemble variance is:
1Var(ensemble) = ρσ² + (1-ρ)/m * σ²Key insight: Bagging reduces variance by factor of m when models are uncorrelated (ρ=0).
Implementing Advanced Bagging:
1from sklearn.ensemble import BaggingClassifier, BaggingRegressor
2from sklearn.tree import DecisionTreeClassifier
3from sklearn.model_selection import cross_val_score
4import numpy as np
5
6# Advanced bagging configuration
7bagging_clf = BaggingClassifier(
8 estimator=DecisionTreeClassifier(
9 max_depth=None, # Deep trees for low bias
10 min_samples_split=2,
11 min_samples_leaf=1,
12 random_state=42
13 ),
14 n_estimators=500, # More trees = better performance
15 max_samples=0.8, # Use 80% of data per tree
16 max_features=0.8, # Feature subsampling
17 bootstrap=True, # Bootstrap sampling
18 bootstrap_features=False, # Don't bootstrap features
19 oob_score=True, # Compute out-of-bag score
20 n_jobs=-1, # Parallel processing
21 random_state=42
22)
23
24# Train and evaluate
25X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
26bagging_clf.fit(X_train, y_train)
27
28print(f"OOB Score: {bagging_clf.oob_score_:.4f}")
29print(f"Test Score: {bagging_clf.score(X_test, y_test):.4f}")Advanced Techniques:
Real-World Applications:
Common Mistakes:
Random forests have numerous hyperparameters that dramatically affect performance. Systematic optimization is essential for production-grade models.
Critical Hyperparameters:
1. n_estimators (Number of Trees):
2. max_depth:
3. min_samples_split:
4. min_samples_leaf:
5. max_features:
Systematic Hyperparameter Optimization:
1from sklearn.ensemble import RandomForestClassifier
2from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
3from scipy.stats import randint, uniform
4import numpy as np
5
6# Define hyperparameter space
7param_dist = {
8 'n_estimators': [100, 200, 500, 1000],
9 'max_depth': [None, 10, 20, 30, 50],
10 'min_samples_split': randint(2, 20),
11 'min_samples_leaf': randint(1, 10),
12 'max_features': ['sqrt', 'log2', 0.3, 0.5, 0.7],
13 'bootstrap': [True, False],
14 'class_weight': ['balanced', None]
15}
16
17# Random search (efficient initial exploration)
18rf_random = RandomizedSearchCV(
19 RandomForestClassifier(random_state=42),
20 param_distributions=param_dist,
21 n_iter=100, # Number of parameter settings sampled
22 cv=5, # Cross-validation folds
23 scoring='roc_auc', # Metric to optimize
24 n_jobs=-1,
25 random_state=42,
26 verbose=2
27)
28
29rf_random.fit(X_train, y_train)
30print(f"Best parameters: {rf_random.best_params_}")
31print(f"Best CV score: {rf_random.best_score_:.4f}")
32
33# Grid search around best parameters (fine-tuning)
34best_params = rf_random.best_params_
35param_grid = {
36 'n_estimators': [best_params['n_estimators'] - 100,
37 best_params['n_estimators'],
38 best_params['n_estimators'] + 100],
39 'max_depth': [best_params['max_depth']],
40 'min_samples_split': [max(2, best_params['min_samples_split'] - 2),
41 best_params['min_samples_split'],
42 best_params['min_samples_split'] + 2],
43 'max_features': [best_params['max_features']]
44}
45
46rf_grid = GridSearchCV(
47 RandomForestClassifier(random_state=42),
48 param_grid=param_grid,
49 cv=5,
50 scoring='roc_auc',
51 n_jobs=-1
52)
53
54rf_grid.fit(X_train, y_train)Advanced Optimization with Optuna:
1import optuna
2from sklearn.ensemble import RandomForestClassifier
3from sklearn.model_selection import cross_val_score
4
5def objective(trial):
6 params = {
7 'n_estimators': trial.suggest_int('n_estimators', 100, 1000, step=100),
8 'max_depth': trial.suggest_int('max_depth', 5, 50),
9 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
10 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
11 'max_features': trial.suggest_categorical('max_features',
12 ['sqrt', 'log2', 0.3, 0.5]),
13 'random_state': 42
14 }
15
16 rf = RandomForestClassifier(**params)
17 score = cross_val_score(rf, X_train, y_train, cv=5,
18 scoring='roc_auc', n_jobs=-1).mean()
19 return score
20
21# Bayesian optimization
22study = optuna.create_study(direction='maximize')
23study.optimize(objective, n_trials=50)
24
25print(f"Best parameters: {study.best_params}")
26print(f"Best score: {study.best_value:.4f}")Performance vs. Speed Tradeoffs:
Hyperparameter Impact on Diversity:
Random forests provide powerful built-in feature importance metrics. Advanced analysis reveals which features truly drive predictions.
Feature Importance Types:
1. Gini Importance (Mean Decrease in Impurity):
Default importance in scikit-learn. Measures average decrease in node impurity when splitting on a feature.
Advantages:
Disadvantages:
2. Permutation Importance (Mean Decrease in Accuracy):
Measures decrease in model performance when feature values are randomly shuffled.
Advantages:
Disadvantages:
Implementing Comprehensive Feature Analysis:
1from sklearn.ensemble import RandomForestClassifier
2from sklearn.inspection import permutation_importance
3import pandas as pd
4import numpy as np
5import matplotlib.pyplot as plt
6
7# Train model
8rf = RandomForestClassifier(n_estimators=500, random_state=42, n_jobs=-1)
9rf.fit(X_train, y_train)
10
11# 1. Gini Importance
12gini_importance = pd.DataFrame({
13 'feature': X_train.columns,
14 'gini_importance': rf.feature_importances_
15}).sort_values('gini_importance', ascending=False)
16
17# 2. Permutation Importance (on validation set)
18perm_importance = permutation_importance(
19 rf, X_val, y_val,
20 n_repeats=10, # Stability
21 random_state=42,
22 n_jobs=-1
23)
24
25perm_importance_df = pd.DataFrame({
26 'feature': X_train.columns,
27 'perm_importance': perm_importance.importances_mean,
28 'perm_std': perm_importance.importances_std
29}).sort_values('perm_importance', ascending=False)
30
31# 3. Feature Importance with Confidence Intervals
32def plot_importance_with_ci(importance_df, title):
33 fig, ax = plt.subplots(figsize=(10, 8))
34
35 importance_df = importance_df.head(20) # Top 20 features
36 y_pos = np.arange(len(importance_df))
37
38 ax.barh(y_pos, importance_df['perm_importance'])
39 ax.errorbar(importance_df['perm_importance'], y_pos,
40 xerr=importance_df['perm_std'],
41 fmt='none', color='red', alpha=0.5)
42
43 ax.set_yticks(y_pos)
44 ax.set_yticklabels(importance_df['feature'])
45 ax.set_xlabel('Importance')
46 ax.set_title(title)
47 ax.grid(True, alpha=0.3)
48 plt.tight_layout()
49 plt.show()
50
51plot_importance_with_ci(perm_importance_df,
52 'Permutation Feature Importance (Top 20)')Advanced Feature Selection:
1from sklearn.feature_selection import SelectFromModel
2
3# Method 1: Threshold-based selection
4selector = SelectFromModel(
5 rf,
6 threshold='median', # Keep features above median importance
7 prefit=True # Use already fitted model
8)
9X_selected = selector.transform(X_train)
10selected_features = X_train.columns[selector.get_support()]
11
12print(f"Selected {len(selected_features)} features:")
13print(selected_features.tolist())
14
15# Method 2: Recursive Feature Elimination with CV
16from sklearn.feature_selection import RFECV
17
18rfecv = RFECV(
19 estimator=RandomForestClassifier(n_estimators=100, random_state=42),
20 step=1, # Remove 1 feature at a time
21 cv=5, # Cross-validation folds
22 scoring='roc_auc',
23 n_jobs=-1
24)
25
26rfecv.fit(X_train, y_train)
27optimal_features = X_train.columns[rfecv.support_]
28
29print(f"Optimal number of features: {rfecv.n_features_}")
30print(f"CV scores: {rfecv.cv_results_['mean_test_score']}")
31
32# Plot feature elimination curve
33plt.figure(figsize=(10, 6))
34plt.plot(range(1, len(rfecv.cv_results_['mean_test_score']) + 1),
35 rfecv.cv_results_['mean_test_score'])
36plt.xlabel('Number of Features Selected')
37plt.ylabel('Cross-Validation Score')
38plt.title('RFECV Performance')
39plt.grid(True, alpha=0.3)
40plt.show()Feature Interaction Analysis:
1# Partial dependence for feature interactions
2from sklearn.inspection import partial_dependence, PartialDependenceDisplay
3
4# Two most important features
5top_features = perm_importance_df.head(2)['feature'].tolist()
6feature_indices = [X_train.columns.get_loc(f) for f in top_features]
7
8# Partial dependence plot
9fig, ax = plt.subplots(figsize=(12, 5))
10PartialDependenceDisplay.from_estimator(
11 rf, X_train, feature_indices,
12 kind='average',
13 ax=ax
14)
15plt.tight_layout()
16plt.show()Best Practices:
Out-of-bag (OOB) estimation provides unbiased performance estimates without separate validation sets. This unique property of bagging enables efficient model evaluation.
Out-of-Bag Theory:
When training with bootstrap sampling, approximately 37% of samples are left out of each tree's training set. These "out-of-bag" samples can be used for validation without any additional computational cost.
How OOB Estimation Works:
Implementing OOB Estimation:
1from sklearn.ensemble import RandomForestClassifier
2import numpy as np
3
4# Enable OOB estimation
5rf_oob = RandomForestClassifier(
6 n_estimators=500,
7 oob_score=True, # Enable OOB calculation
8 bootstrap=True, # Required for OOB
9 random_state=42,
10 n_jobs=-1
11)
12
13rf_oob.fit(X_train, y_train)
14
15# OOB results
16print(f"OOB Score: {rf_oob.oob_score_:.4f}")
17
18# OOB predictions for each sample
19oob_predictions = rf_oob.oob_decision_function_
20print(f"OOB Predictions shape: {oob_predictions.shape}")
21
22# Compare OOB with test set performance
23test_score = rf_oob.score(X_test, y_test)
24print(f"Test Score: {test_score:.4f}")
25print(f"Difference: {abs(rf_oob.oob_score_ - test_score):.4f}")Advanced OOB Analysis:
1from sklearn.metrics import roc_auc_score, roc_curve
2import matplotlib.pyplot as plt
3
4# OOB predictions vs. true labels
5y_oob_pred_proba = rf_oob.oob_decision_function_[:, 1] # Probability of class 1
6oob_auc = roc_auc_score(y_train, y_oob_pred_proba)
7
8# Test set predictions
9y_test_pred_proba = rf_oob.predict_proba(X_test)[:, 1]
10test_auc = roc_auc_score(y_test, y_test_pred_proba)
11
12# ROC curves comparison
13fpr_oob, tpr_oob, _ = roc_curve(y_train, y_oob_pred_proba)
14fpr_test, tpr_test, _ = roc_curve(y_test, y_test_pred_proba)
15
16plt.figure(figsize=(10, 6))
17plt.plot(fpr_oob, tpr_oob, label=f'OOB (AUC = {oob_auc:.4f})', linewidth=2)
18plt.plot(fpr_test, tpr_test, label=f'Test (AUC = {test_auc:.4f})', linewidth=2)
19plt.plot([0, 1], [0, 1], 'k--', label='Random', alpha=0.3)
20plt.xlabel('False Positive Rate')
21plt.ylabel('True Positive Rate')
22plt.title('ROC Curve: OOB vs Test Set')
23plt.legend()
24plt.grid(True, alpha=0.3)
25plt.show()Using OOB for Hyperparameter Tuning:
1# Monitor OOB score during hyperparameter search
2n_estimators_range = [50, 100, 200, 500, 1000]
3max_features_range = ['sqrt', 'log2', 0.3, 0.5, 0.7]
4
5results = []
6
7for n_est in n_estimators_range:
8 for max_feat in max_features_range:
9 rf = RandomForestClassifier(
10 n_estimators=n_est,
11 max_features=max_feat,
12 oob_score=True,
13 random_state=42,
14 n_jobs=-1
15 )
16 rf.fit(X_train, y_train)
17
18 results.append({
19 'n_estimators': n_est,
20 'max_features': max_feat,
21 'oob_score': rf.oob_score_,
22 'test_score': rf.score(X_test, y_test)
23 })
24
25results_df = pd.DataFrame(results)
26best_params = results_df.loc[results_df['oob_score'].idxmax()]
27
28print("Best parameters based on OOB score:")
29print(best_params)Advanced Cross-Validation Strategies:
1from sklearn.model_selection import (
2 cross_val_score,
3 StratifiedKFold,
4 RepeatedStratifiedKFold
5)
6
7# Stratified K-Fold (maintains class distribution)
8skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
9cv_scores = cross_val_score(
10 rf_oob, X_train, y_train,
11 cv=skf,
12 scoring='roc_auc',
13 n_jobs=-1
14)
15
16print(f"Stratified K-Fold CV: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")
17
18# Repeated Stratified K-Fold (more stable estimates)
19rskf = RepeatedStratifiedKFold(
20 n_splits=5,
21 n_repeats=3, # Repeat 3 times
22 random_state=42
23)
24
25repeated_cv_scores = cross_val_score(
26 rf_oob, X_train, y_train,
27 cv=rskf,
28 scoring='roc_auc',
29 n_jobs=-1
30)
31
32print(f"Repeated CV: {repeated_cv_scores.mean():.4f} (+/- {repeated_cv_scores.std():.4f})")OOB vs. Cross-Validation:
Advantages of OOB:
When to Use Cross-Validation:
Best Practices:
Ensemble performance depends critically on model diversity. Advanced techniques maximize diversity while maintaining individual model accuracy.
The Bias-Variance-Diversity Tradeoff:
Ensemble error can be decomposed into:
1Error = Average Individual Error - DiversityKey insight: We want diverse yet accurate base models.
Measuring Ensemble Diversity:
1import numpy as np
2from sklearn.metrics import accuracy_score
3
4def calculate_diversity_metrics(predictions_matrix, y_true):
5 """
6 Calculate ensemble diversity metrics.
7
8 Args:
9 predictions_matrix: (n_samples, n_models) predictions
10 y_true: True labels
11
12 Returns:
13 Dictionary of diversity metrics
14 """
15 n_samples, n_models = predictions_matrix.shape
16
17 # 1. Q-Statistic (pairwise diversity)
18 q_stats = []
19 for i in range(n_models):
20 for j in range(i+1, n_models):
21 pred_i = predictions_matrix[:, i]
22 pred_j = predictions_matrix[:, j]
23
24 # Confusion matrix between two classifiers
25 n11 = np.sum((pred_i == y_true) & (pred_j == y_true))
26 n00 = np.sum((pred_i != y_true) & (pred_j != y_true))
27 n10 = np.sum((pred_i == y_true) & (pred_j != y_true))
28 n01 = np.sum((pred_i != y_true) & (pred_j == y_true))
29
30 # Q-statistic
31 if (n11 * n00 + n01 * n10) != 0:
32 q = (n11 * n00 - n01 * n10) / (n11 * n00 + n01 * n10)
33 q_stats.append(q)
34
35 avg_q = np.mean(q_stats)
36
37 # 2. Disagreement Measure
38 disagreement = []
39 for i in range(n_models):
40 for j in range(i+1, n_models):
41 disagree = np.sum(predictions_matrix[:, i] != predictions_matrix[:, j])
42 disagreement.append(disagree / n_samples)
43
44 avg_disagreement = np.mean(disagreement)
45
46 # 3. Kohavi-Wolpert Variance
47 predictions_vote = np.apply_along_axis(
48 lambda x: np.bincount(x).argmax(),
49 axis=1,
50 arr=predictions_matrix
51 )
52
53 kw_variance = np.mean([
54 np.mean(predictions_matrix[i, :] != predictions_vote[i])
55 for i in range(n_samples)
56 ])
57
58 return {
59 'q_statistic': avg_q,
60 'disagreement': avg_disagreement,
61 'kw_variance': kw_variance
62 }
63
64# Example usage with Random Forest
65from sklearn.ensemble import RandomForestClassifier
66
67rf = RandomForestClassifier(n_estimators=100, random_state=42)
68rf.fit(X_train, y_train)
69
70# Get individual tree predictions
71tree_predictions = np.array([
72 tree.predict(X_test)
73 for tree in rf.estimators_
74]).T
75
76diversity_metrics = calculate_diversity_metrics(tree_predictions, y_test)
77print("Ensemble Diversity Metrics:")
78for metric, value in diversity_metrics.items():
79 print(f" {metric}: {value:.4f}")Advanced Techniques for Increasing Diversity:
1. Feature Subsampling (max_features):
1# Experiment with different max_features values
2max_features_values = ['sqrt', 'log2', 0.3, 0.5, 0.7, 0.9]
3
4for max_feat in max_features_values:
5 rf = RandomForestClassifier(
6 n_estimators=100,
7 max_features=max_feat,
8 random_state=42
9 )
10 rf.fit(X_train, y_train)
11
12 tree_preds = np.array([
13 tree.predict(X_test) for tree in rf.estimators_
14 ]).T
15
16 diversity = calculate_diversity_metrics(tree_preds, y_test)
17 accuracy = accuracy_score(y_test, rf.predict(X_test))
18
19 print(f"max_features={max_feat}:")
20 print(f" Accuracy: {accuracy:.4f}")
21 print(f" Diversity (disagreement): {diversity['disagreement']:.4f}")2. Random Patches (Subsample Features and Samples):
1from sklearn.ensemble import BaggingClassifier
2from sklearn.tree import DecisionTreeClassifier
3
4# Random Patches: subsample both samples and features
5random_patches = BaggingClassifier(
6 estimator=DecisionTreeClassifier(),
7 n_estimators=500,
8 max_samples=0.7, # 70% of samples
9 max_features=0.7, # 70% of features
10 bootstrap=True, # Bootstrap samples
11 bootstrap_features=True, # Bootstrap features
12 random_state=42,
13 n_jobs=-1
14)
15
16random_patches.fit(X_train, y_train)
17print(f"Random Patches Accuracy: {random_patches.score(X_test, y_test):.4f}")3. Extremely Randomized Trees (Extra-Trees):
1from sklearn.ensemble import ExtraTreesClassifier
2
3# Extra-Trees: random thresholds for splits
4extra_trees = ExtraTreesClassifier(
5 n_estimators=500,
6 max_features='sqrt',
7 random_state=42,
8 n_jobs=-1
9)
10
11extra_trees.fit(X_train, y_train)
12
13# Compare with Random Forest
14rf = RandomForestClassifier(
15 n_estimators=500,
16 max_features='sqrt',
17 random_state=42,
18 n_jobs=-1
19)
20rf.fit(X_train, y_train)
21
22print(f"Extra-Trees Accuracy: {extra_trees.score(X_test, y_test):.4f}")
23print(f"Random Forest Accuracy: {rf.score(X_test, y_test):.4f}")
24
25# Compare diversity
26et_preds = np.array([tree.predict(X_test) for tree in extra_trees.estimators_]).T
27rf_preds = np.array([tree.predict(X_test) for tree in rf.estimators_]).T
28
29et_div = calculate_diversity_metrics(et_preds, y_test)
30rf_div = calculate_diversity_metrics(rf_preds, y_test)
31
32print(f"
33Extra-Trees Diversity: {et_div['disagreement']:.4f}")
34print(f"Random Forest Diversity: {rf_div['disagreement']:.4f}")4. Weighted Ensemble:
1# Assign weights based on individual tree performance
2def weighted_ensemble_predict(estimators, X, weights):
3 predictions = np.array([est.predict_proba(X) for est in estimators])
4 weighted_preds = np.average(predictions, axis=0, weights=weights)
5 return np.argmax(weighted_preds, axis=1)
6
7# Calculate weights based on OOB performance
8tree_weights = []
9for tree in rf.estimators_:
10 # Use a subset for validation
11 tree_pred = tree.predict(X_val)
12 tree_acc = accuracy_score(y_val, tree_pred)
13 tree_weights.append(tree_acc)
14
15tree_weights = np.array(tree_weights)
16tree_weights = tree_weights / tree_weights.sum() # Normalize
17
18# Make weighted predictions
19weighted_pred = weighted_ensemble_predict(rf.estimators_, X_test, tree_weights)
20weighted_acc = accuracy_score(y_test, weighted_pred)
21
22print(f"Weighted Ensemble Accuracy: {weighted_acc:.4f}")
23print(f"Standard RF Accuracy: {rf.score(X_test, y_test):.4f}")Best Practices for Maximum Diversity:
Comprehensive demonstration of advanced bagging and random forest techniques including hyperparameter optimization, feature importance analysis, OOB validation, and diversity measurement.
1# Advanced Random Forest Optimization System
2import numpy as np
3import pandas as pd
4from sklearn.datasets import make_classification
5from sklearn.model_selection import train_test_split, cross_val_score
6from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
7from sklearn.inspection import permutation_importance
8from sklearn.metrics import classification_report, roc_auc_score, roc_curve
9import matplotlib.pyplot as plt
10import seaborn as sns
11
12# Set style
13sns.set_style('whitegrid')
14plt.rcParams['figure.figsize'] = (14, 10)
15
16print("=" * 80)
17print("ADVANCED RANDOM FOREST OPTIMIZATION SYSTEM")
18print("=" * 80)
19
20# ============================================================================
21# 1. GENERATE SYNTHETIC DATASET
22# ============================================================================
23print("
241. Generating Synthetic Dataset...")
25print("-" * 80)
26
27X, y = make_classification(
28 n_samples=5000,
29 n_features=30,
30 n_informative=20,
31 n_redundant=5,
32 n_classes=2,
33 weights=[0.7, 0.3], # Imbalanced classes
34 flip_y=0.05, # 5% label noise
35 random_state=42
36)
37
38# Create feature names
39feature_names = [f'feature_{i:02d}' for i in range(X.shape[1])]
40X_df = pd.DataFrame(X, columns=feature_names)
41
42# Split data
43X_train, X_temp, y_train, y_temp = train_test_split(
44 X_df, y, test_size=0.4, random_state=42, stratify=y
45)
46X_val, X_test, y_val, y_test = train_test_split(
47 X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp
48)
49
50print(f"Training set: {X_train.shape}")
51print(f"Validation set: {X_val.shape}")
52print(f"Test set: {X_test.shape}")
53print(f"Class distribution (train): {np.bincount(y_train)}")
54
55# ============================================================================
56# 2. BASELINE MODEL WITH OOB ESTIMATION
57# ============================================================================
58print("
592. Baseline Random Forest with OOB...")
60print("-" * 80)
61
62rf_baseline = RandomForestClassifier(
63 n_estimators=300,
64 oob_score=True,
65 random_state=42,
66 n_jobs=-1
67)
68
69rf_baseline.fit(X_train, y_train)
70
71print(f"OOB Score: {rf_baseline.oob_score_:.4f}")
72print(f"Validation Score: {rf_baseline.score(X_val, y_val):.4f}")
73print(f"Test Score: {rf_baseline.score(X_test, y_test):.4f}")
74
75# ============================================================================
76# 3. HYPERPARAMETER OPTIMIZATION
77# ============================================================================
78print("
793. Hyperparameter Optimization...")
80print("-" * 80)
81
82# Manual grid search with OOB
83param_grid = {
84 'n_estimators': [200, 300, 500],
85 'max_depth': [None, 20, 30],
86 'min_samples_split': [2, 5, 10],
87 'max_features': ['sqrt', 'log2', 0.5]
88}
89
90best_score = 0
91best_params = {}
92
93for n_est in param_grid['n_estimators']:
94 for max_depth in param_grid['max_depth']:
95 for min_split in param_grid['min_samples_split']:
96 for max_feat in param_grid['max_features']:
97 rf = RandomForestClassifier(
98 n_estimators=n_est,
99 max_depth=max_depth,
100 min_samples_split=min_split,
101 max_features=max_feat,
102 oob_score=True,
103 random_state=42,
104 n_jobs=-1
105 )
106 rf.fit(X_train, y_train)
107 val_score = rf.score(X_val, y_val)
108
109 if val_score > best_score:
110 best_score = val_score
111 best_params = {
112 'n_estimators': n_est,
113 'max_depth': max_depth,
114 'min_samples_split': min_split,
115 'max_features': max_feat
116 }
117
118print(f"Best parameters: {best_params}")
119print(f"Best validation score: {best_score:.4f}")
120
121# Train optimized model
122rf_optimized = RandomForestClassifier(
123 **best_params,
124 oob_score=True,
125 random_state=42,
126 n_jobs=-1
127)
128rf_optimized.fit(X_train, y_train)
129
130print(f"
131Optimized Model Performance:")
132print(f" OOB Score: {rf_optimized.oob_score_:.4f}")
133print(f" Validation Score: {rf_optimized.score(X_val, y_val):.4f}")
134print(f" Test Score: {rf_optimized.score(X_test, y_test):.4f}")
135
136# ============================================================================
137# 4. FEATURE IMPORTANCE ANALYSIS
138# ============================================================================
139print("
1404. Feature Importance Analysis...")
141print("-" * 80)
142
143# Gini importance
144gini_importance = pd.DataFrame({
145 'feature': feature_names,
146 'importance': rf_optimized.feature_importances_
147}).sort_values('importance', ascending=False)
148
149# Permutation importance
150perm_importance = permutation_importance(
151 rf_optimized, X_val, y_val,
152 n_repeats=10,
153 random_state=42,
154 n_jobs=-1
155)
156
157perm_importance_df = pd.DataFrame({
158 'feature': feature_names,
159 'importance': perm_importance.importances_mean,
160 'std': perm_importance.importances_std
161}).sort_values('importance', ascending=False)
162
163print("Top 10 Features (Permutation Importance):")
164print(perm_importance_df.head(10))
165
166# ============================================================================
167# 5. ENSEMBLE DIVERSITY ANALYSIS
168# ============================================================================
169print("
1705. Ensemble Diversity Analysis...")
171print("-" * 80)
172
173# Get predictions from individual trees
174tree_predictions = np.array([
175 tree.predict(X_test)
176 for tree in rf_optimized.estimators_
177]).T
178
179# Calculate diversity metrics
180def calculate_disagreement(predictions_matrix):
181 n_samples, n_models = predictions_matrix.shape
182 disagreements = []
183
184 for i in range(n_models):
185 for j in range(i+1, n_models):
186 disagree = np.sum(predictions_matrix[:, i] != predictions_matrix[:, j])
187 disagreements.append(disagree / n_samples)
188
189 return np.mean(disagreements)
190
191diversity_score = calculate_disagreement(tree_predictions)
192print(f"Average Pairwise Disagreement: {diversity_score:.4f}")
193
194# ============================================================================
195# 6. EXTRA-TREES COMPARISON
196# ============================================================================
197print("
1986. Extra-Trees Comparison...")
199print("-" * 80)
200
201extra_trees = ExtraTreesClassifier(
202 **best_params,
203 random_state=42,
204 n_jobs=-1
205)
206extra_trees.fit(X_train, y_train)
207
208et_predictions = np.array([
209 tree.predict(X_test)
210 for tree in extra_trees.estimators_
211]).T
212
213et_diversity = calculate_disagreement(et_predictions)
214
215print(f"Random Forest Test Score: {rf_optimized.score(X_test, y_test):.4f}")
216print(f"Random Forest Diversity: {diversity_score:.4f}")
217print(f"
218Extra-Trees Test Score: {extra_trees.score(X_test, y_test):.4f}")
219print(f"Extra-Trees Diversity: {et_diversity:.4f}")
220
221# ============================================================================
222# 7. VISUALIZATION
223# ============================================================================
224print("
2257. Creating Visualizations...")
226print("-" * 80)
227
228fig = plt.figure(figsize=(16, 12))
229
230# Feature Importance Comparison
231ax1 = plt.subplot(2, 2, 1)
232top_features = perm_importance_df.head(15)
233ax1.barh(range(len(top_features)), top_features['importance'])
234ax1.errorbar(top_features['importance'], range(len(top_features)),
235 xerr=top_features['std'], fmt='none', color='red', alpha=0.5)
236ax1.set_yticks(range(len(top_features)))
237ax1.set_yticklabels(top_features['feature'])
238ax1.set_xlabel('Permutation Importance')
239ax1.set_title('Top 15 Features (Permutation Importance)')
240ax1.grid(True, alpha=0.3)
241
242# ROC Curves
243ax2 = plt.subplot(2, 2, 2)
244y_pred_proba_rf = rf_optimized.predict_proba(X_test)[:, 1]
245y_pred_proba_et = extra_trees.predict_proba(X_test)[:, 1]
246
247fpr_rf, tpr_rf, _ = roc_curve(y_test, y_pred_proba_rf)
248fpr_et, tpr_et, _ = roc_curve(y_test, y_pred_proba_et)
249
250auc_rf = roc_auc_score(y_test, y_pred_proba_rf)
251auc_et = roc_auc_score(y_test, y_pred_proba_et)
252
253ax2.plot(fpr_rf, tpr_rf, label=f'Random Forest (AUC={auc_rf:.4f})', linewidth=2)
254ax2.plot(fpr_et, tpr_et, label=f'Extra-Trees (AUC={auc_et:.4f})', linewidth=2)
255ax2.plot([0, 1], [0, 1], 'k--', alpha=0.3)
256ax2.set_xlabel('False Positive Rate')
257ax2.set_ylabel('True Positive Rate')
258ax2.set_title('ROC Curve Comparison')
259ax2.legend()
260ax2.grid(True, alpha=0.3)
261
262# Learning Curve (n_estimators)
263ax3 = plt.subplot(2, 2, 3)
264n_estimators_range = range(10, 501, 20)
265train_scores = []
266test_scores = []
267
268for n_est in n_estimators_range:
269 rf_temp = RandomForestClassifier(
270 n_estimators=n_est,
271 max_depth=best_params['max_depth'],
272 max_features=best_params['max_features'],
273 random_state=42,
274 n_jobs=-1
275 )
276 rf_temp.fit(X_train, y_train)
277 train_scores.append(rf_temp.score(X_train, y_train))
278 test_scores.append(rf_temp.score(X_test, y_test))
279
280ax3.plot(n_estimators_range, train_scores, label='Training Score', linewidth=2)
281ax3.plot(n_estimators_range, test_scores, label='Test Score', linewidth=2)
282ax3.set_xlabel('Number of Trees')
283ax3.set_ylabel('Accuracy')
284ax3.set_title('Learning Curve: Number of Trees')
285ax3.legend()
286ax3.grid(True, alpha=0.3)
287
288# Diversity vs Performance
289ax4 = plt.subplot(2, 2, 4)
290max_features_values = ['sqrt', 'log2', 0.3, 0.5, 0.7]
291diversities = []
292accuracies = []
293
294for max_feat in max_features_values:
295 rf_temp = RandomForestClassifier(
296 n_estimators=100,
297 max_features=max_feat,
298 random_state=42,
299 n_jobs=-1
300 )
301 rf_temp.fit(X_train, y_train)
302
303 tree_preds = np.array([tree.predict(X_test) for tree in rf_temp.estimators_]).T
304 div = calculate_disagreement(tree_preds)
305 acc = rf_temp.score(X_test, y_test)
306
307 diversities.append(div)
308 accuracies.append(acc)
309
310ax4.scatter(diversities, accuracies, s=100)
311for i, feat in enumerate(max_features_values):
312 ax4.annotate(str(feat), (diversities[i], accuracies[i]),
313 fontsize=9, ha='right')
314ax4.set_xlabel('Ensemble Diversity (Disagreement)')
315ax4.set_ylabel('Accuracy')
316ax4.set_title('Diversity vs. Performance Trade-off')
317ax4.grid(True, alpha=0.3)
318
319plt.tight_layout()
320plt.show()
321
322# ============================================================================
323# FINAL SUMMARY
324# ============================================================================
325print("
326" + "=" * 80)
327print("OPTIMIZATION SUMMARY")
328print("=" * 80)
329print(f"
330Baseline Model:")
331print(f" Test Accuracy: {rf_baseline.score(X_test, y_test):.4f}")
332print(f"
333Optimized Model:")
334print(f" Test Accuracy: {rf_optimized.score(X_test, y_test):.4f}")
335print(f" Improvement: {(rf_optimized.score(X_test, y_test) - rf_baseline.score(X_test, y_test)) * 100:.2f}%")
336print(f"
337Optimal Hyperparameters:")
338for param, value in best_params.items():
339 print(f" {param}: {value}")
340print(f"
341Ensemble Characteristics:")
342print(f" Diversity Score: {diversity_score:.4f}")
343print(f" Number of Trees: {best_params['n_estimators']}")
344print(f" OOB Score: {rf_optimized.oob_score_:.4f}")
345print("
346" + "=" * 80)This comprehensive demonstration showcases advanced random forest and bagging techniques essential for production ML systems:
🎯 Complete Optimization Pipeline:
1. Dataset Preparation:
2. Baseline with OOB:
3. Systematic Hyperparameter Optimization:
4. Feature Importance Analysis:
5. Ensemble Diversity Measurement:
6. Extra-Trees Comparison:
7. Comprehensive Visualizations:
A. Feature Importance Plot:
B. ROC Curve Analysis:
C. Learning Curve:
D. Diversity vs. Performance:
🔬 Advanced Techniques Demonstrated:
Bootstrap Theory Application:
Variance Reduction:
Feature Subsampling:
Model Comparison:
📊 Production Insights:
Performance Metrics:
Optimization Strategy:
Feature Engineering:
Best Practices:
💡 Key Takeaways:
This showcase provides a complete template for production random forest systems with all essential optimization and validation techniques.
Excellent work mastering advanced bagging and random forests! You now understand ensemble theory, hyperparameter optimization, and diversity management. In our next lesson, "Gradient Boosting Mastery (XGBoost, LightGBM, CatBoost)," you'll learn the complementary ensemble technique that builds trees sequentially to correct errors. Get ready to master the most powerful ML algorithms in Kaggle competitions and production systems!
Your progress is saved in this browser, on this device only.