Late Payment Prediction

In the Real World · Brief · Machine learning · Core · 1 to 2 days

Two people can chase 120 invoices a week. Decide which 120. Work it yourself before reading the walkthrough.

The situation

Halloway and Finch is an accountancy practice. Its own sales ledger holds 25,287 invoices across 620 clients and 24 months, and 26.9% of them are paid late. Credit control currently works down the list from the largest invoice.

The partners want a weekly chase list. The constraint is real: 120 calls a week is what two people can do.

The data

One of those columns cannot be used. Working out which, and why, is part of the exercise.

What the partners believe

1. Large invoices are more likely to be paid late.
2. A model that predicts lateness accurately will improve collections.
3. The client’s own history is the strongest signal available.
4. The numbers a classifier outputs can be treated as probabilities.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A model, validated in a way that reflects how it would be used in practice.
3. A ranked chase list of 120 invoices, and a stated reason for the ordering you chose.
4. A comparison of your ordering against at least two alternatives, measured in money rather than in classification metrics.
5. A clear statement of any assumption you had to make that the data does not support.

If you want to go further

• Split the data two ways, once at random and once on time, and compare what each reports.
• Build the simplest possible baseline from a single column before you build anything else, and make your model beat it.
• Work out how the answer changes as the capacity changes from 40 calls a week to 960.
• Decide what you would need to log from now on to stop one of your assumptions being an assumption.

In the Real World · Machine learning · Core · 1 to 2 days

Credit control can chase 120 invoices a week and chases the biggest ones. A model that predicts late payment well makes this worse, because the invoices most likely to go late are the small ones. The fix is one multiplication.

Section 1Problem Definition

No code yet. This project has a capacity constraint in it, and a constraint changes what the model is for. Without one you want good predictions. With one you want a good ordering, and those are not the same objective.

Business objective

Rank this week’s open invoices so the 120 that get chased are the 120 worth chasing. Success is cash brought forward, not invoices correctly labelled.

Success metrics, written before the data is opened

Hypotheses

1. H1. Large invoices are more likely to be paid late.
2. H2. A model that predicts lateness accurately will improve collections.
3. H3. The client’s own history is the strongest signal available.
4. H4. Predicted probabilities can be treated as probabilities.

The assumption that turns risk into cash

A chase brings forward roughly 35% of the value it is aimed at. That number comes from the practice rather than from the data, it is varied in section 7c, and it scales every result without changing any ranking.

Section 2Data Collection

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss

SEED, CAPACITY, RECOVERY = 20260822, 120, 0.35

d = pd.read_csv('data/halloway-finch-ledger.csv')
print('invoices  :', len(d))
print('clients   :', d['client_id'].nunique())
print('late rate :', round(d['paid_late'].mean(), 4))
print('value late:', round(d.loc[d['paid_late'] == 1, 'invoice_amount'].sum()))

invoices  : 25,287
clients   : 620
late rate : 0.2688
value late: 18,732,638

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate rows     :', d.duplicated().sum())
print('duplicate invoices :', d['invoice_id'].duplicated().sum())
print('missing cells      :', d.isna().sum().sum())
print('months             :', d['month_index'].nunique())

duplicate rows     : 0
duplicate invoices : 0
missing cells      : 0
months             : 24

3bHandling categorical mess

print(d.groupby('sector')['paid_late'].agg(['size', 'mean']).round(4)
       .sort_values('mean', ascending=False).to_string())

                        size    mean
sector
Hospitality             3,220  0.3177
Construction            4,819  0.3086
Professional services   5,956  0.2530
Manufacturing           3,930  0.2511
Technology              3,097  0.2486
Retail                  4,265  0.2401

Sector matters: Hospitality runs at 31.8% against Retail at 24.0%. Both are one-hot encoded in 5c rather than ordinal encoded, because there is no order to them.

3cDealing with outliers

print('mean   :', round(d['invoice_amount'].mean(), 2))
print('median :', round(d['invoice_amount'].median(), 2))
print('99th   :', round(d['invoice_amount'].quantile(0.99), 2))
print('max    :', round(d['invoice_amount'].max(), 2))
print('skew   :', round(d['invoice_amount'].skew(), 2))

mean   : 2,796.81
median : 1,491.89
99th   : 18,428.57
max    : 43,189.69
skew   : 3.15

Nothing is trimmed. The large invoices are the entire commercial point of the exercise, and winsorising them would remove exactly the cases the ranking exists to find. This is the opposite of the decision the profit leak audit made about order values, for a different reason: there the tail distorted an average, here the tail is the objective.

3dHandling missing values

missing cells : 0

None. prior_late_rate has no natural value for a client’s first invoice and the ledger carries the practice’s default rather than a null, which is stated in the dictionary and worth knowing: for those rows the feature is an assumption, not a measurement.

3eHandling skewed data

Invoice amount is heavily right-skewed at 3.15. No transform is applied. Gradient boosting splits on order, so the shape is irrelevant to it, and the amount is used unlogged in section 7d because pounds are the unit of the decision. Logging it there would optimise something nobody wants.

3fData types and normalisation

The split is the decision that belongs here, and it is a modelling decision disguised as a preprocessing one.

CUT = 18
tr, te = d[d['month_index'] < CUT], d[d['month_index'] >= CUT]

print(f'train {len(tr):,} invoices, months 0 to {CUT - 1}, late {tr["paid_late"].mean():.4f}')
print(f'test  {len(te):,} invoices, months {CUT} on,     late {te["paid_late"].mean():.4f}')

train 19,044 invoices, months 0 to 17, late 0.271
test  6,243 invoices, months 18 on,     late 0.2622

Section 4Exploratory Data Analysis

4aTarget variable analysis

26.9% of invoices are paid late, carrying 18,732,638 pounds, or 26.5% of billed value. Imbalanced enough that accuracy is useless and balanced enough that nothing exotic is needed.

4bNumerical variables

One relationship decides the shape of this project.

r, p = stats.pointbiserialr(d['paid_late'], d['invoice_amount'])
print(f'correlation between amount and lateness: {r:+.4f}  p {p:.4f}')
print('mean amount, paid late    :', round(d.loc[d['paid_late'] == 1, 'invoice_amount'].mean(), 2))
print('mean amount, paid on time :', round(d.loc[d['paid_late'] == 0, 'invoice_amount'].mean(), 2))

correlation between amount and lateness: -0.0068  p 0.2776
mean amount, paid late    : 2,755.61
mean amount, paid on time : 2,811.96

Late invoices are worth 2,756 pounds on average and on-time invoices 2,812. Size tells you nothing about whether an invoice will be paid on time.

4cCategorical variables

Sector spans 24.0% to 31.8%, and direct debit is the single largest protective factor in the ledger. Both are available at issue.

4dRelationships between variables

The client’s own history is the obvious candidate for the strongest signal, and 7b tests it as a baseline in its own right rather than assuming it.

4eTesting our hypotheses

4fSubgroups

The subgroup that decides the money is invoices in the top decile by amount. They are 10% of the ledger and carry a disproportionate share of the value at risk, and they are no more likely to be late than anything else. Every strategy in 7d is really a statement about how many of them make the list.

Section 5Feature Engineering

5aThe leakage trap

The time split in 3f is the second line of defence. If a leak of that kind survived into the feature table, the gap between random and time-split performance in 7b is where it would show.

5bNew features

Three are constructed at the row level as the ledger is written, and each is a running figure rather than a total: prior invoice count, prior late rate and months since the client last paid late. Nothing is added at modelling time.

5cEncoding

NUM = ['payment_terms_days', 'on_direct_debit', 'invoice_amount', 'prior_invoices',
       'prior_late_rate', 'months_since_last_late', 'disputed_line']
CAT = ['sector', 'client_size']

X = pd.get_dummies(d[NUM + CAT], columns=CAT, drop_first=True)
Xtr, Xte = X.loc[tr.index], X.loc[te.index]
ytr, yte = tr['paid_late'].values, te['paid_late'].values
print('feature matrix :', X.shape)

feature matrix : (25,287, 14)

5dFeature selection

All fourteen columns are kept. With 19,044 training rows there is no pressure to cut, and permutation importance in 7d is used to describe the model rather than to prune it.

Section 6Model Selection

Section 7Model Training

7aBaselines

Ranking on the client’s prior late rate alone reaches 0.6445. Ranking on invoice amount, which is what the practice does today, reaches 0.5101, which is chance. The bar a model has to clear is the history column, not zero.

7bComparing candidates

def boost():
    return HistGradientBoostingClassifier(max_depth=4, learning_rate=0.06,
                                          max_iter=300, random_state=SEED)


def logit():
    return make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=3000, random_state=SEED))


for name, make in (('logistic', logit), ('boosting', boost)):
    m = make().fit(Xtr, ytr)
    p = m.predict_proba(Xte)[:, 1]
    print(f'{name:10s} AUC {roc_auc_score(yte, p):.4f}   '
          f'AP {average_precision_score(yte, p):.4f}   '
          f'Brier {brier_score_loss(yte, p):.5f}')

logistic   AUC 0.6361   AP 0.3678   Brier 0.18553
boosting   AUC 0.6414   AP 0.3802   Brier 0.18411

Boosting wins on average precision and goes forward. Neither model is impressive in absolute terms, and that is the honest position: late payment is substantially driven by things no ledger records, such as whether the client’s own customer paid them this month.

# what a random split would have claimed for the same model
from sklearn.model_selection import train_test_split
Xr_tr, Xr_te, yr_tr, yr_te = train_test_split(X, d['paid_late'].values, test_size=len(te),
                                              random_state=SEED,
                                              stratify=d['paid_late'].values)
m_rand = boost().fit(Xr_tr, yr_tr)
print('random split AUC :', round(roc_auc_score(yr_te, m_rand.predict_proba(Xr_te)[:, 1]), 4))
print('time split AUC   :', 0.6414)

random split AUC : 0.6361
time split AUC   : 0.6414

A difference of 0.0053. On this ledger the random split would not have flattered the model at all, because client behaviour is stable across the two years. Reporting that is more useful than claiming a benefit that did not arrive: the time split is still the right default, and here it cost nothing to be careful.

7cHyperparameter tuning

Almost none, and the reason matters. What is tuned instead is calibration, because section 7d multiplies these probabilities by pounds.

from sklearn.calibration import CalibratedClassifierCV

model = boost().fit(Xtr, ytr)
prob  = model.predict_proba(Xte)[:, 1]

cal      = CalibratedClassifierCV(boost(), method='isotonic', cv=3).fit(Xtr, ytr)
prob_cal = cal.predict_proba(Xte)[:, 1]

print('Brier, raw        :', round(brier_score_loss(yte, prob), 5))
print('Brier, calibrated :', round(brier_score_loss(yte, prob_cal), 5))

bins = pd.qcut(pd.Series(prob).rank(method='first'), 10, labels=False)
for b in sorted(bins.unique()):
    print(f'decile {b + 1:2d}  predicted {prob[bins == b].mean():.4f}  '
          f'actual {yte[bins.values == b].mean():.4f}')

Brier, raw        : 0.18411
Brier, calibrated : 0.18503
decile  1  predicted 0.0586  actual 0.0880
decile  2  predicted 0.1210  actual 0.1667
decile  3  predicted 0.1792  actual 0.1699
decile  4  predicted 0.2352  actual 0.2612
decile  5  predicted 0.2535  actual 0.2528
decile  6  predicted 0.2674  actual 0.2628
decile  7  predicted 0.2927  actual 0.2917
decile  8  predicted 0.3131  actual 0.3237
decile  9  predicted 0.3432  actual 0.3590
decile 10  predicted 0.4067  actual 0.4464

Predicted against actual late rate in each decile. The largest gap is 0.0457, which is close enough that the probabilities can be multiplied by money.

7dFinal evaluation

The model is now scored against the thing it is actually for: a list of 480 invoices, which is about a month of chasing at 120 a week.

te2 = te.copy()
te2['prob']           = prob
te2['expected_value'] = te2['prob'] * te2['invoice_amount']
te2['at_risk']        = te2['paid_late'] * te2['invoice_amount']

K = CAPACITY * 4


def chase_by(col):
    picked = te2.nlargest(K, col)
    return (picked['paid_late'].mean(),
            picked['at_risk'].sum(),
            picked['at_risk'].sum() / te2['at_risk'].sum())


for label, col in [('Biggest invoices first', 'invoice_amount'),
                   ('Highest probability first', 'prob'),
                   ('Highest expected value first', 'expected_value')]:
    prec, val, share = chase_by(col)
    print(f'{label:30s} precision {prec:.3f}   at risk {val:12,.0f}   '
          f'{share:.1%} of all late value')

Biggest invoices first         precision 0.246   at risk    1,560,786    33.6% of all late value
Highest probability first      precision 0.463   at risk      658,854    14.2% of all late value
Highest expected value first   precision 0.329   at risk    1,781,545    38.4% of all late value

The strategy with the best precision finds the least money. Ranking by probability is right 46.2% of the time and recovers 14.2% of the value at risk.

At every capacity the expected value ranking finds more money. Ranking by probability barely beats chasing at random on this measure, which is the whole argument in one line.

What the model is using

The client’s own history dominates, which confirms H3. Invoice amount contributes almost nothing to predicting lateness, and everything to deciding what to do about it.

Section 8Documentation and Handoff

The weekly chase list

What not to do

• Do not report precision as the success measure. The list with the best precision here recovers the least money.
• Do not chase the biggest invoices. It is the current process and the column it ranks on is uncorrelated with lateness.
• Do not rebalance the classes. It would improve the ranking metrics and break the calibration the expected value calculation depends on.
• Do not use days_late as a feature. It is the outcome, and it does not exist when the list is built.
• Do not treat 35% recovery as measured. It is the practice’s estimate. It scales every figure here and changes no ranking, and logging chase outcomes would replace it with a number.

Reproducibility

What to take from this

• A capacity constraint changes the objective from prediction to ranking. Once you can only act on the top N, the question is what N should contain, not how accurate the scores are.
• Rank by what you are optimising. Cash means probability times amount. Probability alone optimises a count.
• Check calibration before you multiply a probability by anything. An uncalibrated score can order a list perfectly well and still be nonsense once it meets a pound sign.
• Split on time when the data has a time index. Here it happened to cost nothing, and you can only know that by doing it.
• Precision and recovery can point in opposite directions. This ledger shows the best precision producing the worst outcome, which is worth remembering the next time a model is signed off on a classification report.

The practice asked for a model that predicts late payment. The model is mediocre, at an AUC of 0.6414, and it still moves 392,942 pounds a month, because the value was never in the prediction. It was in what the prediction gets multiplied by.