Uplift Modelling From an A/B Test

In the Real World · Brief · Machine learning · Advanced · 3 to 5 days

A discount email lifted conversion and finance says it lost money. Work out who should have received it. Do this yourself before reading the walkthrough.

The situation

Northgate Home sent a 12% discount code to half its list and the same email without a code to the other half. 60,000 customers, randomised properly. Conversion in the treated group came out higher.

Finance has looked at the contribution and says the campaign was a loss. Marketing has looked at the conversion rate and says it was a win. You have the file.

The data

The economics

What the room believes

1. The code worked, because conversion went up.
2. Targeting the customers most likely to buy will improve the return.
3. The code helps everyone to some degree, so the worst case of sending it too widely is wasted margin.
4. A model that predicts purchases well will target the code well.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A stated rule, in pounds, for whether an individual customer is worth sending a code to.
3. A model that estimates the effect of the code per customer, and an honest account of how you evaluated it given that the effect is never observed for anyone individually.
4. A comparison of at least three targeting policies, valued in contribution rather than conversions.
5. A list of who should not receive the code, with the evidence that they should not.

If you want to go further

• Check the randomisation before you trust any of it.
• Build a straightforward purchase model as well, then compare how it ranks customers against how your own estimates rank them.
• Work out the break-even condition algebraically before you fit anything, and see whether it changes which customers you would mail.
• Decide what has to be true for this model to still work on the next campaign.

In the Real World · Machine learning · Advanced · 3 to 5 days

The discount email lifted conversion 2.98% points and lost money. Not because the lift was fake, but because most of it went to people who were going to buy anyway. This is the model that tells those two groups apart.

Section 1Problem Definition

No code yet. This project exists because of one distinction, and getting it wrong is the single most expensive mistake in CRM.

The distinction

Almost every targeting model answers the first and is used for the second. The two are different quantities and section 7b shows they barely correlate here.

Four kinds of customer

Business objective

Decide who gets the code next time. The campaign as run had a positive effect on conversion and a negative effect on contribution, so the objective is not more conversions, it is more money.

Hypotheses

1. H1. The code worked, because conversion went up.
2. H2. Targeting the customers most likely to buy will improve the return.
3. H3. The code helps everyone to some degree, so the worst case of sending it too widely is wasted margin.
4. H4. A model that predicts purchases well will target the code well.

The economics, agreed before modelling

Section 2Data Collection

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

SEED = 20260822
MARGIN, DISCOUNT = 0.42, 0.12

d = pd.read_csv('data/northgate-discount-test.csv')
print('rows    :', len(d))
print('treated :', int(d['treated'].sum()), ' control :', int((1 - d['treated']).sum()))
print(d['purchased'].groupby(d['treated']).mean().round(4).to_string())

rows    : 60,000
treated : 30,023  control : 29,977
treated
0    0.1004
1    0.1302

Section 3Data Preprocessing

3aDuplicates and schema checks

On an uplift project the schema check is a validity check, and it comes before everything. If the two arms were not comparable, no amount of modelling recovers a causal effect.

print('duplicate rows      :', d.duplicated().sum())
print('duplicate customers :', d['customer_id'].duplicated().sum())

chi2, p = stats.chisquare([int((1 - d['treated']).sum()), int(d['treated'].sum())])
print(f'split {d["treated"].mean():.5f}   chi2 {chi2:.3f}   p {p:.4f}')

duplicate rows      : 0
duplicate customers : 0
split 0.50038   chi2 0.035   p 0.8510

The split passes at p = 0.851. Worth saying plainly, because the post-test analysis project on this site is entirely about what happens when it does not, and every method below assumes randomisation held.

3bHandling categorical mess

Device and region are clean and get one-hot encoded in 5c. Neither is expected to carry much, and both are kept so the model can prove that rather than have it assumed.

3cDealing with outliers

Order value has the usual right tail. It does not enter the model at all: the outcome being modelled is whether someone bought, and value enters only in the economics in section 7d, as a single average. Modelling the amount as well would be a second project and a second set of assumptions.

3dHandling missing values

missing cells : 0

None. The interesting absence in this dataset is not a null, it is that no column records which of the four customer types anyone is. That is the quantity the whole project has to recover.

3eHandling skewed data

Recency and monetary are both right-skewed. Gradient boosting splits on order rather than on distance so it is untroubled by either, and the logistic baseline in 7b is standardised inside its pipeline. No transform is applied to the shared feature table, because the two models want different things and hiding that in preprocessing would make the comparison dishonest.

3fData types and normalisation

The balance check belongs here, because it is the last thing that can invalidate everything downstream.

FEATS = ['recency_days', 'frequency', 'monetary', 'tenure_days',
         'email_open_rate', 'discount_share_past', 'browsed_last_7d']

for c in FEATS:
    a, b = d.loc[d['treated'] == 1, c], d.loc[d['treated'] == 0, c]
    smd = (a.mean() - b.mean()) / np.sqrt((a.var() + b.var()) / 2)
    print(f'{c:22s} treated {a.mean():9.3f}  control {b.mean():9.3f}  SMD {smd:+.4f}')

recency_days           treated   120.610  control   120.049  SMD +0.0069
frequency              treated     2.612  control     2.623  SMD -0.0066
monetary               treated   189.804  control   189.841  SMD -0.0003
tenure_days            treated   761.036  control   763.738  SMD -0.0064
email_open_rate        treated     0.423  control     0.424  SMD -0.0030
discount_share_past    treated     0.230  control     0.231  SMD -0.0054
browsed_last_7d        treated     0.255  control     0.255  SMD -0.0015

Every standardised mean difference is inside 0.0069, well under the 0.1 anyone would treat as a warning. The randomisation did its job, so any difference in outcome between the arms is caused by the code.

Section 4Exploratory Data Analysis

4aTarget variable analysis

There are two targets here and only one of them is in the data. The observed target is whether someone bought. The target we actually want is the difference the code made to them, which is never observed for any individual.

ct = d.loc[d['treated'] == 1, 'purchased'].mean()
cc = d.loc[d['treated'] == 0, 'purchased'].mean()
se = np.sqrt(ct * (1 - ct) / (d['treated'] == 1).sum()
             + cc * (1 - cc) / (d['treated'] == 0).sum())

print(f'treated {ct:.4f}   control {cc:.4f}')
print(f'average treatment effect {ct - cc:+.4f}  '
      f'95% CI [{ct - cc - 1.96 * se:+.4f}, {ct - cc + 1.96 * se:+.4f}]')

treated 0.1302   control 0.1004
average treatment effect +0.0298  95% CI [+0.0247, +0.0349]

The code lifted conversion by 2.98% points, a relative gain of 29.7%, with a p-value below the smallest number floating point can represent. On any dashboard that is a win.

4bNumerical variables

The features are ordinary CRM history. What matters is not their distributions but whether any of them relates to the difference the code makes, which cannot be seen by looking at one variable at a time. That is why this section is short and section 7 is long.

4cCategorical variables

Device and region are balanced across arms and carry almost nothing. They are left in so the model can say so.

4dRelationships between variables

One relationship can be checked before modelling, and it is the one that sinks H2: the customers most likely to buy are not the ones the code moves.

# how much room is there for a code to add anything, by how likely someone
# already is to buy? Use the CONTROL arm only, so nothing is contaminated.
ctrl = d[d['treated'] == 0].copy()
ctrl['band'] = pd.qcut(ctrl['recency_days'], 5, labels=['most recent', '2', '3', '4',
                                                        'longest ago'])
print(ctrl.groupby('band', observed=True)['purchased'].agg(['size', 'mean']).round(4)
          .to_string())

Split the control arm into deciles of purchase rate and the answer is visible before any uplift model exists. The heaviest buyers convert at several times the base rate with no code at all, so there is very little room for a code to add anything, and every one of them who converts costs 9.32 pounds of margin.

4eTesting our hypotheses

4fSubgroups

The subgroup that matters is the one no column names. Section 7 recovers it, and 7d profiles it, but it is worth stating the shape of the problem now: roughly a quarter of this list is worth a code, and the campaign sent it to everyone.

Section 5Feature Engineering

5aThe leakage trap

The leak in an uplift model is specific and easy to walk into: any feature that knows about the treatment.

The subtler trap is the outcome itself. Two models are fitted, one per arm, and each sees only its own arm’s outcomes. Fitting a single model on everyone with treatment as a feature is the alternative in 6, and it fails for a different reason.

5bNew features

None. Seven behavioural columns and two categoricals, straight from the file.

5cEncoding

FEATS = ['recency_days', 'frequency', 'monetary', 'tenure_days',
         'email_open_rate', 'discount_share_past', 'browsed_last_7d']
CATS  = ['device', 'region']

X = pd.get_dummies(d[FEATS + CATS], columns=CATS, drop_first=True)
y = d['purchased'].values
t = d['treated'].values
print('feature matrix :', X.shape)

feature matrix : (60,000, 14)

5dFeature selection

Everything is kept. With 60,000 rows and fourteen columns there is no dimensionality problem, and dropping features on the basis of how they affect the uplift estimate is circular in the same way that choosing k by the prettiest clusters is circular. There is no held-out truth about individual uplift to validate against.

Section 6Model Selection

How to judge one

Accuracy is meaningless here, because the thing being predicted is never observed. The measure has to be whether ranking by predicted uplift concentrates the real, measured effect at the top.

Section 7Model Training

7aBaselines

Note the order. The propensity model, which is what almost every CRM team would build, is worse than mailing everyone. It finds the sure things and hands each of them 9.32 pounds.

7bComparing candidates

Xtr, Xte, ytr, yte, ttr, tte = train_test_split(
    X, y, t, test_size=0.35, random_state=SEED, stratify=t)


def two_model_uplift(make):
    m_t = make().fit(Xtr[ttr == 1], ytr[ttr == 1])
    m_c = make().fit(Xtr[ttr == 0], ytr[ttr == 0])
    return m_t, m_c, m_t.predict_proba(Xte)[:, 1] - m_c.predict_proba(Xte)[:, 1]


def boosting():
    return HistGradientBoostingClassifier(max_depth=4, learning_rate=0.06,
                                          max_iter=300, random_state=SEED)


m_t, m_c, uplift = two_model_uplift(boosting)
propensity = m_t.predict_proba(Xte)[:, 1]

print('purchase model AUC :', round(roc_auc_score(yte, propensity), 4))
print('uplift and propensity rank correlation :',
      round(stats.spearmanr(uplift, propensity).correlation, 4))

purchase model AUC : 0.6118
uplift and propensity rank correlation : -0.0595

7cHyperparameter tuning

Two model families, each fitted twice, compared on the area under the uplift curve rather than on anything about individual predictions.

Boosting wins on the uplift measure and the two are close on purchase AUC, which is the pattern to expect: they are similarly good at the observable problem and differ on the unobservable one. Boosting is taken forward.

7dFinal evaluation

Does the ranking concentrate the real effect

# Xte keeps the original row labels. Build this with a fresh index and every
# later join back to d quietly lands on the wrong customers.
te = pd.DataFrame({'y': yte, 't': tte, 'uplift': uplift, 'propensity': propensity},
                  index=Xte.index)


def measured_by_decile(col):
    q = pd.qcut(te[col].rank(method='first', ascending=False), 10, labels=False) + 1
    for dec, g in te.groupby(q):
        eff = g.loc[g['t'] == 1, 'y'].mean() - g.loc[g['t'] == 0, 'y'].mean()
        print(f'decile {dec:2d}  n {len(g):5d}  measured uplift {eff:+.4f}')


measured_by_decile('uplift')

decile  1  n  2100  measured uplift +0.0685
decile  2  n  2100  measured uplift +0.0632
decile  3  n  2100  measured uplift +0.0356
decile  4  n  2100  measured uplift +0.0310
decile  5  n  2100  measured uplift +0.0447
decile  6  n  2100  measured uplift +0.0239
decile  7  n  2100  measured uplift +0.0194
decile  8  n  2100  measured uplift +0.0033
decile  9  n  2100  measured uplift +0.0145
decile 10  n  2100  measured uplift -0.0013

The same holdout, ranked two ways, with the real treated-minus-control difference measured inside each decile. Ranking by uplift puts 6.85% points in the top decile. Ranking by purchase probability puts -1.51% there.

The uplift ranking is monotone enough to act on and the bottom decile is negative at -0.0013. Those customers bought less when they got a code, which is H3 refuted: over-sending does not merely waste margin, it can remove sales.

Mail the top 30 per cent by predicted uplift and you capture 55.3% of the total measured effect. Random targeting captures 30 per cent of it, by definition.

What the two groups look like

seen = d.loc[te.index].copy()
seen['uplift'] = te['uplift'].values

hi = seen.nlargest(int(len(seen) * 0.2), 'uplift')
lo = seen.nsmallest(int(len(seen) * 0.2), 'uplift')

for c in FEATS:
    print(f'{c:22s} persuadables {hi[c].mean():9.3f}   bottom {lo[c].mean():9.3f}')

recency_days           persuadables   139.180   bottom    73.637
frequency              persuadables     2.766   bottom     3.201
monetary               persuadables   205.497   bottom   181.339
tenure_days            persuadables   755.285   bottom   768.416
email_open_rate        persuadables     0.505   bottom     0.393
discount_share_past    persuadables     0.393   bottom     0.171
browsed_last_7d        persuadables     0.195   bottom     0.386

What separates the two ends of the uplift ranking. Past discount use and time since the last order point one way, recency and frequency the other.

The persuadables have bought on an offer before, at 0.393 of their past orders against 0.171 for the bottom group, and they have been away longer, at 139 days against 74. The bottom group is recent and frequent: people who were coming back anyway.

The money

AOV  = d.loc[d['purchased'] == 1, 'order_value'].mean()
GAIN = AOV * (MARGIN - DISCOUNT)   # contribution on a sale the code caused
GIVE = AOV * DISCOUNT              # margin handed to someone who would have bought


def policy_value(mask):
    sub = te[mask]
    a, b = sub.loc[sub['t'] == 1, 'y'], sub.loc[sub['t'] == 0, 'y']
    per_head = (a.mean() - b.mean()) * GAIN - b.mean() * GIVE
    return per_head * len(sub), len(sub)


# the rule from section 1, applied per customer rather than per decile
te['baseline_hat']   = m_c.predict_proba(Xte)[:, 1]
te['expected_value'] = te['uplift'] * GAIN - te['baseline_hat'] * GIVE

for label, mask in [
        ('Mail everyone',                    pd.Series(True, index=te.index)),
        ('Top 20% by propensity',            te['propensity'] >= te['propensity'].quantile(0.8)),
        ('Top 20% by uplift',                te['uplift'] >= te['uplift'].quantile(0.8)),
        ('Where expected value is positive', te['expected_value'] > 0)]:
    v, m = policy_value(mask)
    print(f'{label:34s} mailed {m:6,d}   value {v:9,.0f}   per head {v / m:+.3f}')

Mail everyone                      mailed 21,000   value    -5,324   per head -0.254
Top 20% by propensity              mailed  4,209   value    -8,269   per head -1.965
Top 20% by uplift                  mailed  4,200   value     3,621   per head +0.862
Where expected value is positive   mailed 10,204   value     5,282   per head +0.518

Every policy on the same holdout. Targeting by purchase probability is the worst of them, worse than sending to everyone.

Section 8Documentation and Handoff

The two lists

What to do, and who owns it

What not to do

• Do not judge the campaign on conversion. It went up 2.98% points and the campaign lost money.
• Do not target by propensity. It is the intuitive move, it is what a purchase model gives you, and here it is worse than no targeting at all.
• Do not assume the worst case is wasted budget. The bottom decile measures -0.0013: some customers buy less because of the code.
• Do not drop the control group once the model works. It is the only instrument that can measure uplift, and without it the model degrades silently.
• Do not reuse this model for a different offer. The uplift is a property of the offer as much as of the customer.

Reproducibility

What to take from this

• Predicting who buys and predicting who to persuade are different problems. Here they rank-correlate at -0.0595.
• A campaign can lift conversion and lose money. The lift has to clear the margin you hand to everyone who would have bought anyway.
• Write the break-even ratio down before modelling. Uplift has to beat 0.4 times baseline. Everything else is detail.
• Negative uplift is real. Sleeping dogs are not a theoretical category, they are 13.5% of this list.
• Evaluate on measured uplift by decile, on a holdout. There is no individual ground truth, so the only honest test is whether the ranking concentrates the real effect.
• Keep the control group forever. It is the instrument, not the cost.

The campaign looked like a success and was not. What changed the answer was not a better model of who buys, it was asking a different question of the same data: not who converts, but who converts because of us. That question needs a control group, and it is the reason to keep holding one out long after the test is over.