Deep Learning Against Gradient Boosting

In the Real World · Brief · Deep learning · Core · 1 to 2 days

A board member wants to know why the churn model is not deep learning. Answer them properly, in writing, before you read the walkthrough.

The situation

Brightcart Club runs a membership programme. The churn model is live, it is a boosted tree, and it does its job. A board member has read an article and asked why it is not a neural network. You have been asked for a recommendation.

You have 8,550 member records with 27 columns and a ninety day churn flag.

The data

What the room believes

1. A neural network will beat gradient boosting on this data.
2. One train and test split is enough to tell them apart.
3. The architecture is the biggest decision in this pipeline.
4. If deep learning loses, it is because it was not tuned enough.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. At least three model families compared, each with the same tuning budget, and a defence of the budget you chose.
3. A number for the difference between them and an interval around that number.
4. A statement of where the variation in your results actually comes from.
5. The cost of each option, not only its accuracy.
6. A written recommendation the board could read, including the conditions that would change it.

If you want to go further

• Run your comparison ten times on ten splits and look at how often the ranking changes.
• Separate the variation that comes from the model from the variation that comes from the split, and report both.
• Train on a tenth of the data, then a quarter, then all of it, and see whether the curves are converging or diverging.
• Price each option in tuning time, serving complexity and the explanation you could give the retention team.
• Write the recommendation as one paragraph a non-technical director could act on.

In the Real World · Deep learning · Core · 1 to 2 days

A board member asked why the churn model is not deep learning. Three architectures, ten splits, an equal tuning budget each. Logistic regression wins 9 of the 10, and the win is statistically significant. It is also smaller than half the difference between one random split and another, which is the more useful half of the answer.

Section 1Problem Definition

No code yet. This is a question about evidence rather than about architectures, and the reason it is worth a project is that the usual way of answering it cannot distinguish a real difference from a coin toss.

The problem in one sentence

Comparing two models on one train and test split measures the split as much as it measures the models, and nobody reports the second part.

Business objective

Answer the board’s question in writing, with a number attached to the difference, a number attached to the cost, and an explicit list of the conditions under which the recommendation would flip.

Hypotheses

1. H1. A neural network will beat gradient boosting on this data.
2. H2. One train and test split is enough to tell them apart.
3. H3. The architecture is the biggest decision in this pipeline.
4. H4. If deep learning loses, it is because it was not tuned enough.

Section 2Data Collection

import numpy as np
import pandas as pd

raw = pd.read_csv('data/brightcart-members.csv')
print('rows     :', len(raw))
print('columns  :', raw.shape[1])
print('duplicates:', int(raw.duplicated().sum()))
print('churn    :', round(100 * raw['churned_90d'].mean(), 1), 'percent')

rows     : 8550
columns  : 27
duplicates: 50
churn    : 21.6 percent

This is the member file from the published churn code-along, unchanged. 8,550 rows, 27 columns, 1832 churners. Using the same data as the live project is deliberate: the board is asking about the model that already exists, not about a benchmark.

Section 3Data Preprocessing

3aDuplicates and schema checks

d = raw.drop_duplicates().reset_index(drop=True)
print('removed', len(raw) - len(d), 'exact duplicate rows, leaving', len(d))

removed 50 exact duplicate rows, leaving 8500

Duplicates matter more here than usual. A duplicated member can land in both the training and the test set, which inflates every model at once and inflates the flexible ones most.

3bHandling categorical mess

d['country'] = (d['country'].str.strip().str.lower()
                .replace({'uk': 'united kingdom', 'gb': 'united kingdom',
                          'u.k.': 'united kingdom'}))
print('country values before:', raw['country'].nunique(), ' after:', d['country'].nunique())

country values before: 16  after: 10

3cDealing with outliers

bad_age = int(((d['age'] < 16) | (d['age'] > 100)).sum())
bad_min = int((d['avg_session_minutes'] < 0).sum())
d.loc[(d['age'] < 16) | (d['age'] > 100), 'age'] = np.nan
d.loc[d['avg_session_minutes'] < 0, 'avg_session_minutes'] = np.nan
print('impossible ages set to missing      :', bad_age)
print('negative session minutes set missing:', bad_min)

impossible ages set to missing      : 12
negative session minutes set missing: 5

3dHandling missing values

There are 11,602 missing numeric cells. They are filled with the training median, computed inside each split rather than once over the whole file. Computing it once is the quiet version of the leak in section 5a: the test rows contribute to a number the model then uses.

3eHandling skewed data

Spend and ticket counts are heavily skewed. Boosted trees do not care, because a tree splits on order rather than on magnitude. A neural network does care, which is why the numeric columns are standardised before it sees them and are handed to the tree untouched. Treating both the same way would have handicapped one of them and made the comparison worthless.

3fData types and normalisation

NUMS = ['age', 'monthly_fee', 'tenure_months', 'orders_last_90d', 'avg_order_value',
        'total_spend_last_90d', 'days_since_last_order', 'sessions_per_month',
        'avg_session_minutes', 'email_open_rate', 'discount_share_orders',
        'support_tickets_last_6m', 'avg_resolution_hours', 'payment_failures_last_6m',
        'nps_score']
CATS = ['country', 'plan_type', 'payment_method', 'acquisition_channel']
BOOLS = ['auto_renew', 'app_user', 'marketing_opt_in']

def build(with_leak):
    X = d[NUMS].copy()
    for b in BOOLS:
        X[b] = d[b].astype(int)
    if with_leak:
        X['winback_email_sent'] = d['winback_email_sent'].astype(int)
    return pd.concat([X, pd.get_dummies(d[CATS], drop_first=True).astype(float)], axis=1)

print('features without the winback column:', build(False).shape[1])
print('features with it                   :', build(True).shape[1])

features without the winback column: 42
features with it                   : 43

Section 4Exploratory Data Analysis

4aTarget variable analysis

21.6% of members churned in the ninety days after the snapshot. Imbalanced enough that accuracy is useless and mild enough that nothing exotic is needed, so every model below is scored on AUC with average precision and the Brier score alongside it.

4bNumerical variables

There are 15 numeric columns, 4 categorical and 3 boolean. That is a small, wide, entirely tabular problem, which is the setting where the architecture question gets asked most often and answered worst.

4cCategorical variables

Four categorical columns, none with high cardinality. One-hot encoding produces 42 columns in total. This matters for the comparison: entity embeddings, the usual argument for a neural network on tabular data, exist to handle categories with thousands of levels. There are none here, so the strongest card the deep side holds is not in play, and section 8 says so.

4dRelationships between variables

One column behaves unlike anything else in the file.

print('correlation with churn :', round(d['winback_email_sent'].corr(d['churned_90d']), 3))
print('share of members sent one :', round(100 * d['winback_email_sent'].mean(), 1))
print('churn rate, winback sent     :',
      round(100 * d.loc[d['winback_email_sent'], 'churned_90d'].mean(), 1))
print('churn rate, winback not sent :',
      round(100 * d.loc[~d['winback_email_sent'], 'churned_90d'].mean(), 1))

correlation with churn : 0.688
share of members sent one : 19.6
churn rate, winback sent     : 78.8
churn rate, winback not sent : 7.6

A member who received a winback email churned 78.8% of the time against 7.6% for everyone else. The email is not a cause of churn. It is sent to people the business has already decided are leaving, which makes it a record of the answer rather than a clue to it. Section 7a shows what happens if you leave it in.

4eTesting our hypotheses

4fSubgroups

No subgroup analysis changes the answer, and that is worth stating rather than quietly skipping. If a neural network had an advantage it would most likely show on the members with the most behavioural history, and it does not.

Section 5Feature Engineering

5aThe leakage trap

Three leaks are available in this file and only the first is obvious.

5bNew features

None. That is a deliberate choice: adding features would improve every model and confound the only comparison this project exists to make. The point of the exercise is the architecture, so everything else is held still.

5cEncoding

One-hot for the tree and the same one-hot, standardised, for the network. Entity embeddings were tried and are not reported separately because with no high-cardinality column they reduce to a slower one-hot.

5dFeature selection

None beyond dropping the leak. Both model families do their own selection, and removing that difference would be removing part of what is being compared.

Section 6Model Selection

Section 7Model Training

7aBaselines

Here is the bake-off the way it usually gets run: the file as it arrived, one split, one number each.

import time
import torch
import torch.nn as nn
from scipy import stats
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier

torch.set_num_threads(8)
RNG = np.random.default_rng(3)
y = d['churned_90d'].astype(int).values
MODELS = ['Logistic regression', 'Gradient boosting', 'Neural network']


def net_fit(A, ytr, B, cfg, seed=0):
    torch.manual_seed(seed)
    layers, prev = [], A.shape[1]
    for h in cfg['hidden']:
        layers += [nn.Linear(prev, h), nn.ReLU(), nn.Dropout(cfg['drop'])]
        prev = h
    layers.append(nn.Linear(prev, 1))
    net = nn.Sequential(*layers)
    opt = torch.optim.AdamW(net.parameters(), lr=cfg['lr'], weight_decay=cfg['wd'])
    lossf = nn.BCEWithLogitsLoss()
    At = torch.tensor(A, dtype=torch.float32)
    yt = torch.tensor(ytr, dtype=torch.float32)
    for _ in range(cfg['epochs']):
        perm = torch.randperm(len(At))
        for i in range(0, len(At), cfg['batch']):
            idx = perm[i:i + cfg['batch']]
            opt.zero_grad()
            lossf(net(At[idx]).squeeze(1), yt[idx]).backward()
            opt.step()
    net.eval()
    with torch.no_grad():
        return torch.sigmoid(net(torch.tensor(B, dtype=torch.float32)).squeeze(1)).numpy()

# Equal budgets. Both grids are sampled the same number of times, and every
# search happens inside the training folds so the test quarter stays untouched.
NET_GRID = [{'hidden': h, 'drop': dr, 'lr': lr, 'wd': wd, 'epochs': ep, 'batch': bs}
            for h in ([64], [128], [128, 64], [256, 128], [64, 32, 16])
            for dr in (0.0, 0.2, 0.4) for lr in (3e-4, 1e-3, 3e-3)
            for wd in (1e-5, 1e-3) for ep in (30, 80) for bs in (128, 512)]
XGB_GRID = [{'n_estimators': ne, 'max_depth': md, 'learning_rate': lr, 'subsample': ss,
             'colsample_bytree': cs, 'min_child_weight': mc}
            for ne in (150, 400, 900) for md in (2, 3, 4, 6)
            for lr in (0.02, 0.06, 0.15) for ss in (0.7, 1.0)
            for cs in (0.7, 1.0) for mc in (1, 10)]
C_GRID = [0.003, 0.01, 0.03, 0.1, 0.3, 1.0, 3.0, 10.0]
BUDGET = 25
NET_S = [NET_GRID[i] for i in RNG.choice(len(NET_GRID), BUDGET, replace=False)]
XGB_S = [XGB_GRID[i] for i in RNG.choice(len(XGB_GRID), BUDGET, replace=False)]


def one_split(X, seed):
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, stratify=y,
                                          random_state=seed)
    med = Xtr.median()
    sc = StandardScaler().fit(Xtr.fillna(med))
    A, B = sc.transform(Xtr.fillna(med)), sc.transform(Xte.fillna(med))
    folds = list(StratifiedKFold(3, shuffle=True, random_state=0).split(A, ytr))
    out, cost = {}, {}

    t0 = time.time()
    bc = max(C_GRID, key=lambda C: np.mean(
        [roc_auc_score(ytr[v], LogisticRegression(C=C, max_iter=3000).fit(A[t], ytr[t])
                       .predict_proba(A[v])[:, 1]) for t, v in folds]))
    cost['Logistic regression'] = time.time() - t0
    out['Logistic regression'] = (LogisticRegression(C=bc, max_iter=3000).fit(A, ytr)
                                  .predict_proba(B)[:, 1], yte)

    t0 = time.time()
    bx = max(XGB_S, key=lambda c: np.mean(
        [roc_auc_score(ytr[v], XGBClassifier(eval_metric='logloss', random_state=0, **c)
                       .fit(Xtr.iloc[t], ytr[t]).predict_proba(Xtr.iloc[v])[:, 1])
         for t, v in folds]))
    cost['Gradient boosting'] = time.time() - t0
    out['Gradient boosting'] = (XGBClassifier(eval_metric='logloss', random_state=0, **bx)
                                .fit(Xtr, ytr).predict_proba(Xte)[:, 1], yte)

    t0 = time.time()
    bn = max(NET_S, key=lambda c: np.mean(
        [roc_auc_score(ytr[v], net_fit(A[t], ytr[t], A[v], c)) for t, v in folds]))
    cost['Neural network'] = time.time() - t0
    out['Neural network'] = (net_fit(A, ytr, B, bn), yte)
    return out, cost


naive, _ = one_split(build(True), 1)
for m in MODELS:
    p, yte = naive[m]
    print('%-22s AUC %.4f  PR %.4f  Brier %.4f'
          % (m, roc_auc_score(yte, p), average_precision_score(yte, p),
             brier_score_loss(yte, p)))

Logistic regression    AUC 0.9203  PR 0.8106  Brier 0.0816
Gradient boosting      AUC 0.9159  PR 0.7971  Brier 0.0795
Neural network         AUC 0.9177  PR 0.8047  Brier 0.0789

Nothing here is wrong except everything. All three clear 0.9159 AUC, which would be an excellent churn model, and all three are largely reading the winback column identified in section 4d. This is the chart that gets put in the board pack.

7bComparing candidates

Drop the leaked column and run the same three models on 10 different random splits, tuning each one inside the training folds every time. This is the part that takes minutes rather than seconds, and it is the only part that answers the question.

X = build(False)
rows, costs = [], []
for seed in range(1, 11):
    res, cost = one_split(X, seed)
    for m in MODELS:
        rows.append({'seed': seed, 'model': m,
                     'auc': roc_auc_score(res[m][1], res[m][0])})
        costs.append({'model': m, 'search_s': cost[m]})
piv = pd.DataFrame(rows).pivot(index='seed', columns='model', values='auc')
costs = pd.DataFrame(costs)

for m in MODELS:
    print('%-22s mean AUC %.4f  sd %.4f  range %.4f to %.4f'
          % (m, piv[m].mean(), piv[m].std(), piv[m].min(), piv[m].max()))

Logistic regression    mean AUC 0.7585  sd 0.0131  range 0.7368 to 0.7807
Gradient boosting      mean AUC 0.7541  sd 0.0127  range 0.7319 to 0.7785
Neural network         mean AUC 0.7527  sd 0.0130  range 0.7332 to 0.7744

Three bars you would struggle to tell apart. The best and worst mean are 0.0058 apart, which is smaller than the standard deviation of any single one of them, and the simplest model is on top.

This is the chart that answers the board’s question. The ranking is not stable: 9 splits to logistic regression and 1 splits to gradient boosting. Whoever ran the comparison once and reported a winner reported which split they happened to draw.

7cHyperparameter tuning

print(costs.groupby('model')['search_s'].mean().round(2))

Logistic regression    search     0.05s   final fit   0.001s   predict  0.08ms   most often 0.01
Gradient boosting      search    24.00s   final fit   0.112s   predict  3.31ms   most often depth 2, 150 trees
Neural network         search    24.62s   final fit   0.209s   predict  0.15ms   most often 64, 30 epochs

The tuning search costs 24.62 seconds for the network and 0.05 for the logistic regression, a factor of 492.0, for a difference in AUC of 0.0058. On this dataset that is seconds. On a dataset a hundred times larger it is the difference between a coffee and an afternoon.

7dFinal evaluation

Three questions decide the recommendation. Is any difference real, where does the variation actually come from, and would more data change it.

for a, b in [('Neural network', 'Gradient boosting'),
             ('Neural network', 'Logistic regression'),
             ('Gradient boosting', 'Logistic regression')]:
    diff = piv[a] - piv[b]
    t, pv = stats.ttest_rel(piv[a], piv[b])
    half = 2.262 * diff.std() / np.sqrt(len(diff))
    print('%-20s minus %-20s %+.4f  [%+.4f, %+.4f]  p %.3f'
          % (a, b, diff.mean(), diff.mean() - half, diff.mean() + half, pv))

print()
print('spread between splits, same model : %.4f' % piv.std().mean())
print('spread between models, same split : %.4f' % piv.std(axis=1).mean())

Neural network       minus Gradient boosting    -0.0014  [-0.0036, +0.0007]  p 0.170
Neural network       minus Logistic regression  -0.0059  [-0.0073, -0.0044]  p 0.000
Gradient boosting    minus Logistic regression  -0.0044  [-0.0065, -0.0024]  p 0.001

spread between splits, same model : 0.0129
spread between models, same split : 0.0034

Is any difference real

Paired across the same 10 splits, which removes the split-to-split variation that dominates everything else and is what makes a difference this small measurable at all. Two of the three comparisons come back significant: logistic regression really does beat both the tree and the network on this data, by 0.0059 and 0.0044 AUC.

Where the variation comes from

This is the finding. Which architecture you pick moves AUC by 0.0034. Which quarter of the data happens to land in your test set moves it by 0.0129. Any bake-off run on a single split is reporting the second number and calling it the first.

Would more data change it

The standard defence is that neural networks need more data. Over a tenfold increase in training rows the network gains 0.04 AUC and the boosted tree gains 0.0728. The network is not on a steeper curve here, and all three are flattening. That is evidence about this dataset at this size, not a general law, and section 8 says what would change it.

Section 8Documentation and Handoff

What to do, and who owns it

What would change the answer

• An input a tree cannot read. Support ticket text, session sequences, or images. That is a different problem and deep learning wins it outright.
• A categorical with thousands of levels. Entity embeddings earn their place there. This file has none.
• Far more rows. Not ten times more, which is measured above and changes nothing. Two or three orders of magnitude.
• A pretrained model to start from. Transfer is where the advantage actually lives, and there is nothing to transfer from on a bespoke member table.
• A multi-task or multi-output target. One network predicting several related outcomes shares structure that separate trees cannot.

What not to do

• Do not report a winner from one split. The ordering here changes with the split.
• Do not confuse a significant difference with a difference worth acting on. Two of the three comparisons here clear p equals 0.05 and none of them would change a decision.
• Do not leave a column in because it helps. The winback column helps more than anything else in the file and it is worthless.
• Do not answer a board question with a benchmark. They asked about your model on your data, and a public leaderboard cannot answer that.
• Do not conclude that deep learning is overrated. It is being asked to do the one job it has no advantage at.

Reproducibility

What to take from this

• On tabular data of this size, the architecture is not the decision. Three families, equal budgets, 0.0058 AUC between them, and the oldest method on top.
• A single split cannot tell two models apart. Its noise is 3.78 times the effect being measured.
• Pair the comparison and report the interval. It costs nothing and it is the difference between a finding and an anecdote.
• One leaked column outweighs every modelling choice. Look for it before arguing about layers.
• When accuracy ties, cost decides. Tuning time, serving complexity and explainability are the real differences here.
• Name the conditions that would flip your recommendation. It is the part that makes a written answer worth keeping.

The board asked a reasonable question and the useful answer is not yes or no. It is that the difference they are asking about is smaller than the difference between two arbitrary ways of cutting the same file, and that the column nobody asked about is worth more than all of it. The recommendation is to keep the tree. The finding is that the question was aimed at the wrong part of the pipeline.