Ensemble Methods in Python

A single model has a ceiling. It carries its own biases and blind spots, and no amount of tuning fully removes them. Ensemble methods break through that ceiling with a deceptively simple idea: combine several models so the group performs better than any individual member. What separates the four major families is not the idea but the mechanics, how the models are trained, how their predictions are combined, and which weakness the combination fixes. Voting and averaging combine independent models by consensus. Bagging trains the same model on many resampled datasets to cut variance. Boosting trains models in sequence so each fixes the last one’s mistakes, cutting bias. And stacking trains a second model to learn the best way to combine the first layer. This article works through all four with runnable code.

The math behind this: https://datalad.co.uk/the-mathematics-behind-gradient-boosting-and-ensembles/

A quick map before the details:

The rule of thumb for choosing: reach for voting when you want a quick win from models you already trust, bagging when one strong model overfits, boosting when your models underfit, and stacking when you have diverse models and want the combination itself to be learned.

The Baseline: A Single Decision Tree

Decision trees are the building blocks of most ensembles, so understanding how one behaves motivates everything that follows.

from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
tree = DecisionTreeRegressor(min_samples_leaf=3, min_samples_split=9, random_state=500)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
print('MAE: {:.3f}'.format(mean_absolute_error(y_test, y_pred)))

A decision tree is a flowchart of yes-or-no questions about features until it reaches an answer. The min_samples_leaf=3constraint stops it basing a final answer on fewer than three training rows, which prevents memorising single outliers, and min_samples_split=9 stops it from splitting groups smaller than nine. Mean absolute error reports the average miss in the target’s own units, which makes it directly interpretable and less sensitive to occasional large errors than RMSE.

Voting and Averaging

The simplest ensemble trains several different algorithms on the same data and combines their outputs. There are two flavours. Hard voting takes the majority of the predicted class labels. Soft voting averages the predicted probabilities and then picks the class with the highest average. Soft voting is usually better when every base model can produce probabilities, because it keeps the confidence information that a hard vote throws away.

Before combining anything, measure each model alone, especially on imbalanced data where accuracy lies. Imagine a fraud classifier where only about eight percent of transactions are fraudulent.

from sklearn.metrics import f1_score
score_lr = f1_score(y_test, clf_lr.predict(X_test))
score_dt = f1_score(y_test, clf_dt.predict(X_test))
score_knn = f1_score(y_test, clf_knn.predict(X_test))

The F1-score is the right metric here because a model that never predicts fraud would score over ninety percent accuracy yet zero F1, instantly exposing the problem. These individual scores also tell you whether an ensemble can help: if all three models score alike, they are probably making the same mistakes, and combining them adds little.

A hard-voting ensemble combines three models with genuinely different worldviews.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import VotingClassifier
clf_knn = KNeighborsClassifier(n_neighbors=5)
clf_lr = LogisticRegression(class_weight="balanced")
clf_dt = DecisionTreeClassifier(min_samples_leaf=3, min_samples_split=9, random_state=500)
clf_vote = VotingClassifier(estimators=[('knn', clf_knn), ('lr', clf_lr), ('dt', clf_dt)])
clf_vote.fit(X_train, y_train)

KNN reasons by proximity, logistic regression draws a linear boundary, and the tree follows a flowchart of rules. The VotingClassifier runs all three on each example and takes the majority. The class_weight="balanced" setting tells logistic regression to treat the rare fraud class as if it appeared proportionally more often, otherwise it learns to mostly predict the majority and becomes accurate but useless. Evaluating with both f1_score and classification_report then shows not just whether the ensemble improved overall, but whether it improved on the hard, rare class specifically.

Soft voting requires every model to produce probabilities, which means a small adjustment for support vector machines.

from sklearn.svm import SVC
clf_svm = SVC(probability=True, class_weight='balanced', random_state=500)
estimators = [('lr', clf_lr), ('dt', clf_dt), ('svm', clf_svm)]
clf_avg = VotingClassifier(estimators, voting='soft')
clf_avg.fit(X_train, y_train)

An SVM normally just outputs a class with no confidence attached, so probability=True runs an internal calibration that converts its decision scores into genuine probabilities, at the cost of slower training. With voting='soft', the models’ probabilities are averaged per class and the highest wins, so a model that says ninety-five percent is treated differently from one that says fifty-one, a distinction a hard vote cannot make. Running both strategies on the same models is a clean controlled experiment: the comparison tells you whether confidence information actually helps for your data, and sometimes the classes are clear enough that the two tie.

Bagging

Bagging, short for bootstrap aggregating, trains the same algorithm on many bootstrap samples of the training set, drawn with replacement, then averages the results. It targets variance, the main weakness of high-capacity models like deep trees. The counterintuitive insight is that deliberately weakening each tree, then combining many de-correlated weak trees, produces a strong and stable ensemble.

The mechanics are clearest done by hand. A single bootstrap sample looks like this.

X_sample = X_train.sample(frac=1.0, replace=True, random_state=42)
y_sample = y_train.loc[X_sample.index]

Sampling with replacement at full size is like drawing training rows from a hat and replacing each before the next draw. Some rows appear several times and some never appear at all, and on average about sixty-three percent of rows show up while the other thirty-seven percent are left out. Those left-out rows form a free validation set, a point we return to shortly. Looping this many times and voting across the resulting trees is bagging in its raw form, and sklearn packages it directly.

from sklearn.ensemble import BaggingClassifier
clf_dt = DecisionTreeClassifier(max_depth=4)
clf_bag = BaggingClassifier(base_estimator=clf_dt, n_estimators=21, random_state=500)
clf_bag.fit(X_train, y_train)

The base_estimator is a blueprint that gets cloned twenty-one times, each clone trained on its own bootstrap sample, with the majority vote returned at prediction time. An odd number of estimators avoids ties. The free validation set from those out-of-bag rows is available with a single flag.

clf_bag = BaggingClassifier(base_estimator=clf_dt, n_estimators=21, oob_score=True, random_state=500)
clf_bag.fit(X_train, y_train)
print('OOB score: {:.3f}'.format(clf_bag.oob_score_))

Because roughly thirty-seven percent of trees never saw any given training row, the oob_score_ predicts each row using only the trees that excluded it, giving a nearly unbiased estimate of test performance at zero extra cost and without touching the test set. It is cross-validation hiding inside the bagging process.

One more parameter turns plain bagging into something more powerful.

clf_lr = LogisticRegression(class_weight='balanced', solver='liblinear', random_state=42)
clf_bag = BaggingClassifier(base_estimator=clf_lr, max_features=10, oob_score=True, random_state=500)
clf_bag.fit(X_train, y_train)

The max_features=10 setting gives each base model a different random subset of features. Without it, every base model sees all features, becomes highly correlated with its peers, makes the same mistakes, and the averaging barely helps. By restricting features, each model develops a partial but distinct perspective, and together they cover the full feature space through complementary blind spots. This is exactly the trick that distinguishes a Random Forest from plain bagging, applied there to trees.

Boosting

Boosting flips the strategy from parallel to sequential. Each new model focuses on the errors its predecessors made, and the final prediction is a weighted combination, which reduces bias rather than variance. The idea is visible in a hand-built version.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
model_one = LinearRegression()
model_one.fit(X_train, y_train)
residuals = model_one.predict(X_train) - y_train  # what model one got wrong
model_two = LinearRegression()
model_two.fit(X_train_alt, residuals)             # learn the leftover error

Model one learns the main signal, the residuals capture what it could not explain, and model two learns to predict those residuals. The final prediction is model one’s answer plus model two’s correction. Gradient boosting automates exactly this across dozens or hundreds of sequential models.

The two classic boosting schemes differ in how they direct attention. AdaBoost reweights samples.

from sklearn.ensemble import AdaBoostRegressor
reg_ada = AdaBoostRegressor(n_estimators=12, random_state=500)
reg_ada.fit(X_train, y_train)

After each of the twelve rounds, AdaBoost increases the weight of the examples it predicted poorly, forcing the next model to concentrate on the hard cases, and the final answer is a weighted average where better-performing rounds earn a larger vote. Omitting base_estimator defaults to a shallow decision tree, which interacts better with the reweighting scheme than a linear model because it can capture the non-linear patterns the reweighting gradually surfaces.

Gradient boosting directs attention differently, fitting each new tree to the gradient of the loss.

from sklearn.ensemble import GradientBoostingClassifier
clf_gbm = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, random_state=500)
clf_gbm.fit(X_train, y_train)

Instead of reweighting samples, it asks for each example which direction the prediction should move to reduce the loss, fits a tree to those gradients, and steps in that direction by the learning rate. One trade-off governs all boosting: a lower learning rate makes each correction smaller and safer but requires more estimators to compensate, so the two parameters trade off directly. Tuning that balance is covered in the hyperparameter tuning guide.

Beyond sklearn, a family of optimised libraries dominates tabular competitions. XGBoost is a heavily regularised, parallelised gradient booster, covered in depth in the XGBoost article. LightGBM grows trees leaf by leaf rather than level by level, reaching low error faster on large data at some overfitting risk on small data. CatBoost handles categorical features natively, with no manual encoding required.

import lightgbm as lgb
import xgboost as xgb
reg_xgb = xgb.XGBRegressor(max_depth=3, learning_rate=0.1, n_estimators=100,
                           objective='reg:squarederror', random_state=500)
reg_lgb = lgb.LGBMRegressor(max_depth=3, learning_rate=0.1, n_estimators=100,
                            objective='mean_squared_error', seed=500)

Watch for the library-specific naming gotchas: LightGBM uses seed where others use random_state, and the same squared-error loss has different string names across libraries. As a quick guide, reach for sklearn’s gradient booster on small simple problems, CatBoost when you have categorical features, LightGBM when you need speed on large data, and XGBoost when you want maximum tunability.

Stacking

Stacking is the most sophisticated family. It trains diverse base models in a first layer, then feeds their predictions as input features to a meta-model in a second layer that learns the optimal way to combine them. Where voting uses a fixed rule, stacking learns the rule, which lets it discover patterns like “trust model A more when this feature is large” that a fixed vote can never capture. The cost is a risk of overfitting if the layers are not separated carefully with cross-validation.

from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import StackingClassifier
base_models = [
    ('knn', KNeighborsClassifier(n_neighbors=5, algorithm='ball_tree')),
    ('dt', DecisionTreeClassifier(min_samples_leaf=5, min_samples_split=15, random_state=500)),
    ('nb', GaussianNB())
]
clf_stack = StackingClassifier(
    estimators=base_models,
    final_estimator=LogisticRegression(),
    stack_method='predict_proba',
    passthrough=False
)
clf_stack.fit(X_train, y_train)

Four algorithms here reason in completely different ways: KNN by proximity, the tree by rules, Naive Bayes by Bayes’ theorem assuming feature independence, and the meta logistic regression by a learned linear boundary over their outputs. The crucial detail sklearn handles automatically is that during fitting it trains the base models with cross-validation, so each training example gets a first-layer prediction from a model that never saw it. Without that, the meta-model would learn spurious signals from the base models’ overfitting. Setting stack_method='predict_proba' passes probability distributions rather than hard labels, giving the meta-model richer information, and passthrough=False means the meta-model sees only the base outputs, not the original features.

Always evaluate the base models alone first, because stacking should beat the best of them; if it does not, the meta-model is adding nothing. The same architecture exists in the mlxtend library with different parameter names, where classifiersreplaces estimators, meta_classifier replaces final_estimator, 

use_probas=True replaces stack_method='predict_proba', and use_features_in_secondary replaces passthrough. Because both packages export a class called StackingClassifier, it is worth confirming which one you imported. Stacking works for regression too, where three diverse regressors such as a tree, plain linear regression, and ridge feed a linear meta-model that learns how much to trust each.

Choosing an Ensemble

Match the method to the symptom in your single model. If it overfits, with a large gap between train and test performance, bagging or a Random Forest averages out the variance. If it underfits, with high error on both, boosting sequentially reduces the bias. If you already have several decent but different models, voting or averaging is the cheap combination. If you want the combination itself to be learned, stacking lets a meta-model handle the weighting. And for tabular data specifically, CatBoost shines with categorical features while LightGBM wins on speed at scale.

The Pitfalls That Recur

A handful of mistakes undermine ensembles. Using soft voting when a base model cannot produce probabilities errors out, which is why SVMs need probability=True. Judging imbalanced data by accuracy hides a model that simply predicts the majority class, so use F1 instead, and remember class_weight='balanced' so models do not ignore the rare class. Building a custom stacking setup without cross-validation lets the first layer overfit and teaches the meta-model the wrong combination, which is exactly the trap sklearn’s StackingClassifier avoids internally. Bagging trees that are too shallow gives the averaging nothing strong to work with, while boosting with too many estimators at a high learning rate overfits. And the deepest principle of all: adding correlated base models that make the same mistakes brings no benefit, so diversity across genuinely different algorithm families is what makes an ensemble worth building.

Conclusion

Ensembles beat single models by combining them, and the four families do it differently. Voting and averaging pool independent models by consensus, with soft voting preserving confidence that hard voting discards. Bagging trains one algorithm on many bootstrap samples to cut variance, and feature subsampling plus tree base learners turns it into a Random Forest, with out-of-bag scoring as a free validation bonus. Boosting trains models in sequence to cut bias, trading learning rate against the number of estimators, and the optimised libraries XGBoost, LightGBM, and CatBoost are its production-grade form. Stacking trains a meta-model to learn the combination, which is the most powerful approach when your base models are diverse and you protect against leakage with cross-validation. Diagnose whether your model overfits or underfits, pick the family that targets that weakness, and always build your ensemble from models different enough to be wrong in different ways.