Lead Scoring and Threshold Economics

In the Real World · Brief · Machine learning · Core · 1 to 2 days

Sales wants a lead score so they can work the best leads first. Build it, then work out how far down the list anyone should go. Do it yourself before reading the walkthrough.

The situation

Vantith is a B2B SaaS business. 34,000 inbound leads over 52 weeks, 6.4% of which convert at a mean deal of 12,941 pounds. Reps currently work whatever arrived most recently.

The data

One of those columns will improve your model and must not be in it. Working out which, and why, is part of the exercise.

The economics

What the room believes

1. Sales should work the highest scoring leads first.
2. Every inbound lead is worth working, because the deals are large.
3. A better model is the biggest improvement available here.
4. How quickly a lead is contacted is an operational detail, not a modelling concern.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A scoring model that could actually be run at the moment a lead arrives.
3. A stated rule, in pounds, for how far down the ranked list to work, and the arithmetic behind it.
4. A comparison of your rule against working everything, measured in contribution.
5. A recommendation ranked by size, so the business knows which change to make first.

If you want to go further

• Fit the model twice, once with every column and once with only what is knowable on arrival, and compare.
• Plot contribution against how much of the list you work, and find where it peaks.
• Quantify the biggest non-model lever you can find in the data, and compare it to what your model is worth.
• Say which of your findings would survive being called causal, and which needs a test.

In the Real World · Machine learning · Core · 1 to 2 days

Sales wanted a lead score so they could work the best leads first. The score is worth 1.82 times the base conversion rate. Answering the phone sooner is worth 5.36 times. The model is the smaller half of the answer.

Section 1Problem Definition

No code yet. There are two questions hiding in a lead scoring request and only one of them is a modelling question.

Sales asked for the first. The second is where the money is, and it needs numbers the model cannot supply.

The economics, agreed before modelling

Hypotheses

1. H1. Sales should work the highest scoring leads first.
2. H2. Every inbound lead is worth working, since the deals are large and the cost of a call is not.
3. H3. A better model is the biggest available improvement.
4. H4. How quickly a lead is contacted is an operational detail, not a modelling concern.

Section 2Data Collection

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss

SEED, COST_TO_WORK, GROSS_MARGIN = 20260822, 450.0, 0.78

d = pd.read_csv('data/vantith-leads.csv')
print('leads      :', len(d))
print('conversion :', round(d['converted'].mean(), 4))
print('mean deal  :', round(d.loc[d['converted'] == 1, 'deal_value'].mean()))

leads      : 34,000
conversion : 0.0641
mean deal  : 12,941

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate rows :', d.duplicated().sum())
print('duplicate ids  :', d['lead_id'].duplicated().sum())
print('missing cells  :', d.isna().sum().sum())

duplicate rows : 0
duplicate ids  : 0
missing cells  : 0

3bHandling categorical mess

print(d.groupby('source')[['converted']].agg(['size', 'mean']).round(4).to_string())

                        size    mean
source
Partner referral        3,385  0.1383
Webinar                 4,391  0.0720
Organic                 7,088  0.0628
Paid search             8,176  0.0544
Cold outbound reply     3,441  0.0477
Content download        7,519  0.0456

Partner referral converts at 13.8% and Content download at 4.6%. Source, company size and role are all one-hot encoded in 5c.

3cDealing with outliers

Deal values have the long right tail every B2B book has. They do not enter the model, which predicts whether a lead converts rather than for how much, and the economics in 7d use the mean deal as a single figure. Section 7d also varies it, because a mean over a skewed distribution is exactly the assumption worth testing.

3dHandling missing values

missing cells : 0

None. Worth one line.

3eHandling skewed data

Response time is right-skewed: a median of 35.7 hours against a mean of 45.3. That gap is the operational story, and section 4b measures what it costs.

3fData types and normalisation

LEAD = ['pages_viewed', 'emails_opened', 'work_email', 'phone_given', 'demo_requested']
CAT  = ['source', 'company_size', 'job_role']

X_lead = pd.get_dummies(d[LEAD + CAT], columns=CAT, drop_first=True)

# the same matrix with the thing that has not happened yet added, for section 5a
X_all = X_lead.copy()
X_all['first_touch_hours'] = d['first_touch_hours'].values

y = d['converted'].values
print('usable features :', X_lead.shape[1])
print('with the leak   :', X_all.shape[1])

usable features : 15
with the leak   : 16

Section 4Exploratory Data Analysis

4aTarget variable analysis

y = d['converted'].values
mean_margin = d.loc[d['converted'] == 1, 'deal_value'].mean() * GROSS_MARGIN
break_even  = COST_TO_WORK / mean_margin

print('conversion            :', round(d['converted'].mean(), 4))
print('contribution per win  :', round(mean_margin))
print('break-even probability:', round(break_even, 5))
print('leads below break-even at the base rate:',
      'the average lead clears it' if d['converted'].mean() > break_even else 'most do not')

conversion            : 0.0641
contribution per win  : 10,094
break-even probability: 0.04458
leads below break-even at the base rate: the average lead clears it

6.4% of leads convert. Against a break-even of 4.46% that means the average lead is worth working and a good share of them are not, which is the entire threshold question in one sentence.

4bNumerical variables

One numerical column dominates everything else in this dataset, and it is the one that is not allowed in the model.

for lo, hi in [(0, 6), (6, 24), (24, 72), (72, 168), (168, 400)]:
    g = d[(d['first_touch_hours'] >= lo) & (d['first_touch_hours'] < hi)]
    if len(g) < 30:
        continue
    print(f'{lo:3d} to {hi:3d} hours  n {len(g):6,d}  conversion {g["converted"].mean():.4f}')

  0 to   6 hours  n  2,045  conversion 0.1051
  6 to  24 hours  n  9,491  conversion 0.0828
 24 to  72 hours  n 15,928  conversion 0.0599
 72 to 168 hours  n  6,179  conversion 0.0354
168 to 400 hours  n    357  conversion 0.0196

A lead contacted inside six hours converts at 10.5%. The same kind of lead contacted after a week converts at 2.0%.

4cCategorical variables

The waits are broadly similar across sources, which is itself a finding: the queue is worked in the order things arrive, not in the order they matter.

4dRelationships between variables

The relationship that matters is between response time and conversion, and it is already established. The second is between company size and deal value, which is why section 7d checks whether the threshold should differ by segment rather than being one number for the whole list.

4eTesting our hypotheses

4fSubgroups

Large companies convert at similar rates to small ones and are worth several times as much when they do. That makes the break-even probability different for them, and section 7d returns to it: one threshold for the whole list is a simplification, and a defensible one only while the segments are close.

Section 5Feature Engineering

5aThe leakage trap

This dataset contains a column that improves the model and destroys it. Fit both and look at the difference.

tr_idx, te_idx = train_test_split(d.index, test_size=0.3, random_state=SEED, stratify=y)


def boost():
    return HistGradientBoostingClassifier(max_depth=4, learning_rate=0.06,
                                          max_iter=300, random_state=SEED)


for name, Xuse in [('lead attributes only', X_lead), ('including response time', X_all)]:
    m = boost().fit(Xuse.loc[tr_idx], y[tr_idx])
    p = m.predict_proba(Xuse.loc[te_idx])[:, 1]
    print(f'{name:26s} AUC {roc_auc_score(y[te_idx], p):.4f}   '
          f'AP {average_precision_score(y[te_idx], p):.4f}')

lead attributes only       AUC 0.6844   AP 0.1330
including response time    AUC 0.7032   AP 0.1557

The tell is not statistical, it is a question about time: was this knowable when the prediction is needed? Ask it of every column and this one answers no.

5bNew features

None. Everything in the file that is legitimately available is already in a usable form, and the temptation worth resisting is engineering something out of first_touch_hours that smuggles it back in.

5cEncoding

Source, company size and role are one-hot encoded. Company size is ordered and could have been encoded as an integer, which would tell a linear model something true and tell a tree nothing it cannot find on its own. It is left as dummies so the two model families in 7b are compared on the same matrix.

5dFeature selection

Fifteen columns, all kept. The selection decision that matters in this project is which single column to exclude, and 5a makes it.

Section 6Model Selection

Section 7Model Training

7aBaselines

7bComparing candidates

Boosting reaches an AUC of 0.6844 against 0.6915 for logistic regression on the same columns. The gap is small. Boosting is taken forward on average precision, and the honest note is that a logistic model would have been defensible and easier to explain to a sales floor.

7cHyperparameter tuning

Left at sensible defaults. What is checked instead is whether the probabilities can be compared against a break-even figure.

model = boost().fit(X_lead.loc[tr_idx], y[tr_idx])
prob  = model.predict_proba(X_lead.loc[te_idx])[:, 1]

print('Brier :', round(brier_score_loss(y[te_idx], prob), 5))
bins = pd.qcut(pd.Series(prob).rank(method='first'), 10, labels=False)
for b in sorted(bins.unique()):
    print(f'decile {b + 1:2d}  predicted {prob[bins == b].mean():.4f}  '
          f'actual {y[te_idx][bins.values == b].mean():.4f}')

Brier : 0.05843
decile  1  predicted 0.0182  actual 0.0206
decile  2  predicted 0.0275  actual 0.0225
decile  3  predicted 0.0361  actual 0.0255
decile  4  predicted 0.0428  actual 0.0520
decile  5  predicted 0.0484  actual 0.0559
decile  6  predicted 0.0551  actual 0.0480
decile  7  predicted 0.0634  actual 0.0667
decile  8  predicted 0.0750  actual 0.0814
decile  9  predicted 0.0998  actual 0.1157
decile 10  predicted 0.1830  actual 0.1529

Largest gap between predicted and actual in any decile is 0.0301, which is close enough to compare against a break-even of 4.46%.

7dFinal evaluation

How far down the list to work

mean_margin = d.loc[d['converted'] == 1, 'deal_value'].mean() * GROSS_MARGIN
break_even  = COST_TO_WORK / mean_margin
print('contribution per win :', round(mean_margin))
print('break-even probability:', round(break_even, 5))

te = d.loc[te_idx].copy()
te['prob'] = prob

for q in [0.0, 0.2, 0.3, 0.4, 0.5, 0.6]:
    sub = te[te['prob'] >= te['prob'].quantile(q)]
    profit = sub['converted'].sum() * mean_margin - len(sub) * COST_TO_WORK
    print(f'work the top {1 - q:4.0%}  leads {len(sub):6,d}  '
          f'wins {int(sub["converted"].sum()):4d}  profit {profit:12,.0f}')

contribution per win : 10,094
break-even probability: 0.04458
work the top  100%  leads 10,200  wins  654  profit    2,011,314
work the top   80%  leads  8,160  wins  610  profit    2,485,189
work the top   70%  leads  7,140  wins  584  profit    2,681,751
work the top   60%  leads  6,120  wins  531  profit    2,605,782
work the top   50%  leads  5,100  wins  474  profit    2,489,439
work the top   40%  leads  4,085  wins  426  profit    2,461,688

Profit peaks at 70.0% of the list and falls away on both sides. Working everything leaves 670,437 pounds on the table.

# the rule from section 1, applied per lead rather than as a percentage cut
te['expected_profit'] = te['prob'] * mean_margin - COST_TO_WORK
worked = te[te['expected_profit'] > 0]

profit = worked['converted'].sum() * mean_margin - len(worked) * COST_TO_WORK
print(f'rule works {len(worked):,} of {len(te):,} leads '
      f'({len(worked) / len(te):.1%})   profit {profit:,.0f}')

rule works 6,327 of 10,200 leads (62.0%)   profit 2,613,570

Selection against speed

The model multiplies conversion by 1.82 on its top tier. Answering sooner multiplies it by 5.36.

If every lead were contacted inside six hours, conversion across the whole book would move from 6.4% to roughly 10.5%. That is about 1,392 extra deals a year and 14,054,541 pounds of contribution, from process rather than from prediction.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not put response time in the model. It improves AUC by 0.0188 and cannot be known when a score is needed.
• Do not set the threshold as a top percentage. 70.0% is the right cut for these economics and for no others.
• Do not work every lead because the deals are large. That is H2, and it costs about 670,437 pounds on this holdout.
• Do not treat the speed finding as settled. Part of it is selection. The test that separates them takes a month.
• Do not use one threshold for every segment once deal sizes diverge further. Large accounts have a lower break-even because a win is worth more.

Reproducibility

What to take from this

• A lead scoring request contains two questions. Which leads, and how many. Only the first is a model and only the second was worth 670,437 pounds here.
• Ask of every column whether it exists at prediction time. Response time passes every statistical check and fails that one.
• A small AUC gain from a leaky feature is more dangerous than a large one. 0.0188 looks like a fair feature.
• Set thresholds from economics, not from percentiles. Break-even here is 4.46%, and it is a fact about the business rather than about the data.
• Check whether the biggest lever is a model at all. Here it was not, and the analysis that found that took less time than the model did.
• Say which of your findings is causal and which is not. The threshold is arithmetic. The speed number needs a test.

Sales asked for a score so they could work the best leads first. They should work fewer leads, sooner. The model is a real improvement and it is the smaller half of the answer, which is worth saying out loud in the handoff rather than letting the model be the headline because it was the thing that was asked for.