XGBoost: 10 Code-Along Examples

The XGBoost article explains why this library keeps winning on tabular data: gradient boosting builds trees in sequence, each one correcting the errors of the ensemble so far, and XGBoost wraps that idea in regularisation, clever sampling, and speed. This workbook runs the full workflow: the scikit-learn interface, the native DMatrix API, cross-validation, early stopping, the regularisation dials, tree visualisation, and a leak-free tuned pipeline to finish. The article’s framing is worth keeping in view throughout: XGBoost is the tool for tabular problems with enough rows, and the boosting rounds dial is the one that overfits, which is why early stopping appears in the middle of this workbook rather than the end.

1. Your first XGBClassifier, against a baseline

XGBoost speaks scikit-learn’s language: create, fit, predict, score. The honest first step is to run it beside a single decision tree, so the ensemble has to prove it earns its complexity.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
# the baseline: one tree
tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
print("single tree accuracy:", round(tree.score(X_test, y_test), 3))
# the ensemble: many trees, each correcting the last
xgb_clf = XGBClassifier(n_estimators=100, random_state=42, eval_metric="logloss")
xgb_clf.fit(X_train, y_train)
print("XGBoost accuracy:    ", round(xgb_clf.score(X_test, y_test), 3))

The interface is identical to every scikit-learn model, which is deliberate: XGBClassifier drops into any existing workflow, pipeline, or grid search unchanged. The comparison is the point of the example. A single tree memorises its training data and wobbles on the test set, while a hundred boosted trees, each a small correction to the ensemble before it, generalise better, and printing both scores side by side shows the gap the ensemble buys. If the gap were zero, the article’s advice would apply: XGBoost is not the tool for every dataset, and the baseline just told you so.

2. Regression with XGBRegressor

Swap the class and the objective, and the same machinery predicts numbers instead of labels. RMSE, the typical error in the target’s own units, is the score to watch.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from xgboost import XGBRegressor
X, y = load_diabetes(return_X_y=True)   # predict disease progression
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
xgb_reg = XGBRegressor(
    objective="reg:squarederror",   # the regression loss
    n_estimators=100,
    random_state=42,
)
xgb_reg.fit(X_train, y_train)
predictions = xgb_reg.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, predictions))
print("RMSE:", round(rmse, 1))
print("first 3 predictions vs actual:")
for p, a in zip(predictions[:3], y_test[:3]):
    print(f"  predicted {p:6.1f}   actual {a:6.1f}")

The objective argument names the loss being boosted: reg:squarederror for regression, binary:logistic for the classifier in Example 1, and each objective changes what the trees correct. RMSE answers the practical question, “how far off is a typical prediction”, in the same units as the target, which makes it reportable to someone who has never heard of boosting. The mechanics underneath are unchanged from classification: trees built in sequence, each fitted to the ensemble’s current errors.

3. The native API and DMatrix

Under the scikit-learn wrapper sits XGBoost’s own API. DMatrix is its optimised data container, and xgb.train takes a parameter dictionary rather than constructor arguments. You will meet this dialect in most XGBoost documentation, so it pays to read both.

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=42, stratify=y)
# the native data container: data and labels together
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
params = {
    "objective": "binary:logistic",   # predict a probability
    "max_depth": 4,
    "eta": 0.1,                       # the learning rate, native name
}
booster = xgb.train(params, dtrain, num_boost_round=100)
probabilities = booster.predict(dtest)          # probabilities, not labels
predictions = (probabilities > 0.5).astype(int)
accuracy = (predictions == y_test).mean()
print("native API accuracy:", round(accuracy, 3))

Three translations to note between the dialects. Data goes into a DMatrix, which stores features and labels together in XGBoost’s internal format and is what makes the native API fast. Hyperparameters travel in a dictionary, with some renamed: the learning rate is eta here and learning_rate in the wrapper. And predict returns raw probabilities under binary:logistic, so the thresholding to labels is yours to do, which is a feature rather than a nuisance, since the threshold is a business decision as the churn article argued.

4. Cross-validation with xgb.cv

The native API’s xgb.cv runs k-fold cross-validation inside XGBoost itself, returning a per-round table of train and test scores that shows the ensemble improving, and then overfitting, as rounds accumulate.

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
dtrain = xgb.DMatrix(X, label=y)
params = {"objective": "binary:logistic", "max_depth": 4, "eta": 0.1}
results = xgb.cv(
    params, dtrain,
    num_boost_round=200,
    nfold=5,                      # 5-fold cross-validation
    metrics="auc",
    seed=42,
)
print(results.tail(3))            # one row per boosting round
best_round = results["test-auc-mean"].idxmax()
print(f"\nbest test AUC {results['test-auc-mean'].max():.4f} "
      f"at round {best_round}")

The returned frame is a round-by-round history: each row holds the mean and standard deviation of the train and test AUC across the five folds after that many trees. Reading it tells you the story a single fit hides, since the train column keeps climbing forever while the test column rises, plateaus, and eventually slips back as later trees start fitting noise. The idxmax line finds the round where test performance peaked, which is exactly the number the next example automates.

5. Early stopping

Rather than guessing the number of rounds, let the validation set decide. With early_stopping_rounds, training halts when the score has not improved for that many rounds, and the best iteration is kept.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
model = XGBClassifier(
    n_estimators=1000,              # a deliberately huge ceiling
    learning_rate=0.1,
    early_stopping_rounds=15,       # stop after 15 rounds without improvement
    eval_metric="auc",
    random_state=42,
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],      # the data that decides when to stop
    verbose=False,
)
print("rounds actually trained:", model.best_iteration + 1)
print("best validation AUC:   ", round(model.best_score, 4))

The ceiling of 1,000 trees is never reached: training watches the validation AUC after every round and stops once fifteen rounds pass without a new best, keeping the model from the best round rather than the last. This is the practical answer to the overfitting curve Example 4 exposed, and it changes how you tune. The number of trees stops being a hyperparameter to search and becomes an outcome, discovered automatically, which frees your tuning budget for the dials that early stopping cannot set, the ones in the next two examples.

6. Regularisation: lambda, alpha, and gamma

Three parameters price complexity inside the objective itself. reg_lambda shrinks leaf values smoothly, reg_alpha pushes weak ones to zero, and gamma charges a toll per split, refusing splits that do not pay.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
print("param        value   CV AUC")
for name, values in [("reg_lambda", [0, 1, 10, 100]),
                     ("reg_alpha",  [0, 1, 10, 100]),
                     ("gamma",      [0, 1, 5, 25])]:
    for v in values:
        model = XGBClassifier(
            n_estimators=100, random_state=42,
            eval_metric="logloss", **{name: v},
        )
        score = cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
        print(f"{name:11s}  {v:5}   {score:.4f}")
    print()

The three act at different points in tree building, echoing the gradient boosting math article. reg_lambda is L2 shrinkage on leaf values, biting hardest on leaves backed by little data. reg_alpha is L1, capable of zeroing a leaf’s contribution entirely. And gamma acts earlier, at split time: a split must improve the loss by more than gamma or it is refused, which means large values prune the trees as they grow. The printed grid shows each dial’s effect on cross-validated AUC, and on a clean dataset like this the differences are small, which is itself the lesson: regularisation matters most when data is noisy or scarce, and the defaults are sensible until the validation curve says otherwise.

7. Tree depth and learning rate: the core trade

The two most consequential dials are max_depth, how complex each tree may be, and learning_rate, how much of each tree’s correction is applied. They trade against the number of rounds, and against each other.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print("depth  lr     rounds used   val AUC")
for depth in [2, 4, 8]:
    for lr in [0.3, 0.1, 0.03]:
        model = XGBClassifier(
            n_estimators=2000, max_depth=depth, learning_rate=lr,
            early_stopping_rounds=20, eval_metric="auc", random_state=42,
        )
        model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
        print(f"  {depth}    {lr:4}   {model.best_iteration + 1:6d}       "
              f"{model.best_score:.4f}")

Two patterns emerge from the grid. Lower learning rates need more rounds, visibly: at 0.03 the model trains hundreds of trees where 0.3 stops after a few dozen, because each tree contributes a smaller step and the ensemble must travel the same distance, exactly the step-size reading from the math article. And shallow trees with a low rate often match or beat deep trees with a high one, because many small careful corrections compound better than a few aggressive ones. The standard recipe follows: fix a low-ish learning rate, let early stopping set the rounds, and spend your search budget on depth and the sampling dials.

8. Sampling: subsample and colsample_bytree

Borrowing from random forests, XGBoost can build each tree on a random fraction of the rows and a random fraction of the columns, which decorrelates the trees and resists overfitting.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
print("subsample  colsample  CV AUC")
for sub in [1.0, 0.8, 0.6]:
    for col in [1.0, 0.8, 0.6]:
        model = XGBClassifier(
            n_estimators=100,
            subsample=sub,             # fraction of ROWS per tree
            colsample_bytree=col,      # fraction of COLUMNS per tree
            random_state=42, eval_metric="logloss",
        )
        score = cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
        print(f"   {sub:3}       {col:3}     {score:.4f}")

Each tree seeing only, say, 80 percent of the rows and 80 percent of the columns learns a slightly different view of the data, and the ensemble of varied trees generalises better than an ensemble of near-clones, the same decorrelation logic that max_features serves in a random forest. The grid shows mild sampling usually matching or beating none, with very aggressive sampling starting to cost accuracy as each tree sees too little. Values between 0.6 and 0.9 for both are the conventional search range, and on larger datasets sampling also buys a real speed-up, since every tree touches less data.

9. Seeing the model: trees and feature importance

Two built-in plots open the black box a crack: plot_tree draws an individual tree’s decisions, and plot_importance ranks features by how much the whole ensemble used them.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
dtrain = xgb.DMatrix(data.data, label=data.target,
                     feature_names=list(data.feature_names))
params = {"objective": "binary:logistic", "max_depth": 3, "eta": 0.1}
booster = xgb.train(params, dtrain, num_boost_round=50)
# one tree from the ensemble, drawn as a diagram
fig, ax = plt.subplots(figsize=(16, 6))
xgb.plot_tree(booster, num_trees=0, ax=ax)      # the FIRST tree
plt.savefig("xgb_tree.png", dpi=150, bbox_inches="tight")
# which features did the whole ensemble lean on?
fig, ax = plt.subplots(figsize=(8, 6))
xgb.plot_importance(booster, max_num_features=10, ax=ax)
plt.savefig("xgb_importance.png", dpi=150, bbox_inches="tight")
print("saved xgb_tree.png and xgb_importance.png")

The two plots answer different questions. The tree diagram shows one member of the ensemble in full: each node a feature threshold, each leaf a contribution to the prediction, which makes the “sequence of small correctors” idea concrete, though remember it is one tree of fifty, not the model. The importance chart aggregates across all trees, counting how often each feature was chosen to split, and it is the fastest sanity check XGBoost offers: if a feature you know should matter ranks nowhere, or an ID-like column dominates, the plot has caught a data problem before the model ships.

10. The capstone: a leak-free tuned pipeline

The finale assembles the article’s workflow advice: preprocessing and model in one pipeline, hyperparameters searched with RandomizedSearchCV, and the test set untouched until the single final verdict.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score
from xgboost import XGBClassifier
X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(0)
X_messy = X.copy()
X_messy[rng.random(X.shape) < 0.05] = np.nan     # 5% holes, as real data has
X_train, X_test, y_train, y_test = train_test_split(
    X_messy, y, random_state=42, stratify=y)
pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("xgb", XGBClassifier(random_state=42, eval_metric="logloss")),
])
search = RandomizedSearchCV(
    pipe,
    {
        "xgb__n_estimators": [100, 200, 400],
        "xgb__max_depth": [2, 3, 4, 6],
        "xgb__learning_rate": [0.03, 0.1, 0.3],
        "xgb__subsample": [0.6, 0.8, 1.0],
        "xgb__colsample_bytree": [0.6, 0.8, 1.0],
    },
    n_iter=20, scoring="roc_auc", cv=5, random_state=42, n_jobs=-1,
).fit(X_train, y_train)
print("best params:", search.best_params_)
print("best CV AUC:", round(search.best_score_, 4))
# the test set appears exactly once, at the very end
proba = search.best_estimator_.predict_proba(X_test)[:, 1]
print("test AUC:   ", round(roc_auc_score(y_test, proba), 4))

Every discipline from the series meets here. The imputer lives inside the pipeline, so its medians are recomputed on each fold’s training portion and the cross-validation stays honest, which is the article’s line about preprocessing inside the CV loop being the difference between an honest number and a flattering one. RandomizedSearchCV samples twenty configurations rather than exhausting the grid, the right trade per the hyperparameter search math. And the test AUC is computed once, at the end, on rows no fit or search ever saw. A worthwhile note for practice: XGBoost can actually handle NaN natively by learning a default direction per split, but the pipeline pattern shown here generalises to every preprocessing step that cannot.

Work through these and you have the whole article in practice: both APIs, classification and regression objectives, xgb.cv‘s round-by-round history, early stopping, the three regularisation dials, the depth-and-learning-rate trade, row and column sampling, the two diagnostic plots, and the tuned leak-free pipeline. The tuning order the article recommends is the one to remember: let early stopping set the rounds, tune depth and the sampling fractions, lower the learning rate as the final polish, and reach for the regularisation terms when the validation curve, not habit, says the model is overfitting.

See you soon, Andrei.