Anomaly Detection When Positives Are Rare

In the Real World · Brief · Machine learning · Core · 1 to 2 days

Two analysts can review 200 card payments a day. Decide which ones. Work it yourself before reading the walkthrough.

The situation

Merrow Pay is a small card acquirer. You have 180,000 authorisations with what the risk engine could see at the time, and a label added months later when a chargeback confirmed fraud. Fraud is rare.

The review team currently works down from the largest payment. They want a better queue, and they cannot review more than 200 cases a day.

The data

What the room believes

1. Unusually large payments are the fraudulent ones.
2. Anomaly detection is the right tool for finding fraud.
3. A gradient boosting model will beat a logistic regression, as it usually does.
4. A review queue that is mostly false positives is a failed model.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A ranked queue of the size the team can actually work, and a stated reason for where you cut it.
3. A comparison against what the team does today, in frauds caught rather than in classification metrics.
4. A defence of whichever metric you chose to report, and a statement of why the obvious one is not on your list.
5. A recommendation for what should happen to the cases the team reviews and clears.

If you want to go further

• Fit something that uses no labels at all, and compare it with something that does. The gap between them is what the labels are worth.
• Work out what your precision would need to be beaten against before you would call it good.
• Plot how much fraud you would catch as the review team grows, so the conversation about headcount has a number in it.
• Decide what you would do about the fact that the label arrives up to 120 days after the transaction.

In the Real World · Machine learning · Core · 1 to 2 days

A model that flags nothing at all is 99.56% accurate. The review team can look at 200 cases a day and currently looks at the biggest payments, which finds almost nothing. This is how you build a queue worth working.

Section 1Problem Definition

No code yet. Everything about this problem is distorted by one number: 0.438% of transactions are fraud. Every default in every library is wrong at that rate, and the first job is to say why.

What the rarity breaks

Business objective

Fill a review queue of 200 cases a day with the cases most likely to be fraud. Success is fraud caught per analyst hour, not a classification report.

Hypotheses

1. H1. Unusually large payments are the fraudulent ones.
2. H2. Anomaly detection is the right tool for finding fraud.
3. H3. A gradient boosting model will beat a logistic regression, as it usually does.
4. H4. A review queue that is mostly false positives is a failed model.

Section 2Data Collection

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, IsolationForest
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import (roc_auc_score, average_precision_score, accuracy_score,
                             confusion_matrix)

SEED, CAPACITY = 20260822, 200

d = pd.read_csv('data/merrow-pay-transactions.csv')
print('transactions :', len(d))
print('fraud        :', int(d['is_fraud'].sum()), f"({d['is_fraud'].mean():.4%})")
print('accuracy of flagging nothing :', round(1 - d['is_fraud'].mean(), 6))

transactions : 180,000
fraud        : 788 (0.4378%)
accuracy of flagging nothing : 0.995622

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate rows :', d.duplicated().sum())
print('duplicate ids  :', d['txn_id'].duplicated().sum())
print('missing cells  :', d.isna().sum().sum())

duplicate rows : 0
duplicate ids  : 0
missing cells  : 0

3bHandling categorical mess

One categorical, the merchant category, one-hot encoded in 5c. Nothing to clean.

3cDealing with outliers

This is the section that answers H1, and it is the reason unsupervised anomaly detection is not automatically the right tool.

p99 = d['amount'].quantile(0.99)
print('99th percentile amount :', round(p99, 2))
print('fraud rate above it    :', round(d.loc[d['amount'] > p99, 'is_fraud'].mean(), 5))
print('fraud rate overall     :', round(d['is_fraud'].mean(), 5))
print('lift                   :',
      round(d.loc[d['amount'] > p99, 'is_fraud'].mean() / d['is_fraud'].mean(), 2))

99th percentile amount : 542.12
fraud rate above it    : 0.00778
fraud rate overall     : 0.00438
lift                   : 1.78

The largest one per cent of payments are 1.78 times more likely to be fraud than average. On a base rate of 0.438% that is almost nothing.

Nothing is trimmed. Removing the tail would remove the point.

3dHandling missing values

missing cells : 0

None, which is normal for an authorisation log: the fields either arrive with the message or the transaction is declined.

3eHandling skewed data

Amount is heavily skewed and amount_ratio, the payment against the card’s own ninety day average, is the more useful version of it. Both are kept. No transform is applied for the tree, and the logistic model standardises inside its pipeline.

3fData types and normalisation

NUM = ['hour_of_day', 'amount', 'avg_amount_90d', 'amount_ratio', 'card_age_days',
       'txns_last_24h', 'distinct_mcc_24h', 'country_mismatch', 'cvv_absent', 'new_device']
CAT = ['mcc']

X = pd.get_dummies(d[NUM + CAT], columns=CAT, drop_first=True)
y = d['is_fraud'].values
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=SEED, stratify=y)

print('train', len(Xtr), ' test', len(Xte), ' fraud in test', int(yte.sum()))

train 126,000  test 54,000  fraud in test 236

Section 4Exploratory Data Analysis

4aTarget variable analysis

788 frauds in 180,000 transactions, worth 66,730 pounds out of 12,818,565. One positive for every 228 rows.

print('fraud       :', int(d['is_fraud'].sum()), f"({d['is_fraud'].mean():.4%})")
print('one in      :', round(1 / d['is_fraud'].mean()))
print('fraud value :', round(d.loc[d['is_fraud'] == 1, 'amount'].sum()),
      'of', round(d['amount'].sum()))

fraud       : 788 (0.4378%)
one in      : 228
fraud value : 66,730 of 12,818,565

At that rate the majority class is so dominant that most defaults quietly optimise for it. Section 7a makes that concrete rather than leaving it as a warning.

4bNumerical variables

Amount is dealt with in 3c. The more useful numerical signals are the authorisation flags, and they are binary rather than continuous, which is why the categorical section below carries more weight than usual.

4cCategorical variables

Three binary flags do most of the work: whether the billing country matched, whether the CVV was supplied, and whether the device was new. They are the fields a risk engine exists to read, and their value is not in any one of them.

for cm in (0, 1):
    for cv in (0, 1):
        g = d[(d['country_mismatch'] == cm) & (d['cvv_absent'] == cv)]
        print(f'country_mismatch {cm}  cvv_absent {cv}  n {len(g):6,d}  '
              f'fraud {g["is_fraud"].mean():.5f}  lift {g["is_fraud"].mean() / d["is_fraud"].mean():5.2f}x')

country_mismatch 0  cvv_absent 0  n 159,871  fraud 0.00208  lift  0.48x
country_mismatch 0  cvv_absent 1  n 14,617  fraud 0.01334  lift  3.05x
country_mismatch 1  cvv_absent 0  n  5,032  fraud 0.01749  lift  3.99x
country_mismatch 1  cvv_absent 1  n    480  fraud 0.35833  lift 81.85x

4dRelationships between variables

Velocity and card age interact in the same way: a burst of transactions on a card opened last week is a different thing from the same burst on a card held for three years. Both are in the feature set and neither is engineered into an explicit interaction, deliberately, so section 7b compares a model that can find them against one that cannot.

4eTesting our hypotheses

4fSubgroups

Gaming and electronics carry the highest fraud rates and the lowest volumes, which is the usual shape. No segment is excluded: a queue built only on high risk categories would miss the fraud that deliberately hides in ordinary ones.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

One, and it was built into the file rather than added at modelling time: amount_ratio, the payment divided by the card’s own ninety day average. It is the difference between knowing a payment is large and knowing it is large for this card, and the second is the question fraud detection actually asks.

5cEncoding

Merchant category one-hot encoded, everything else already numeric or binary. No target encoding: at 0.438% a category mean is computed on a handful of positives and leaks the target straight into the feature.

5dFeature selection

Everything is kept. The temptation with rare events is to prune aggressively because the model looks unstable, and the instability is the sample size rather than the feature count.

Section 6Model Selection

How they are judged

Section 7Model Training

7aBaselines

sup = HistGradientBoostingClassifier(max_depth=4, learning_rate=0.06, max_iter=400,
                                     random_state=SEED).fit(Xtr, ytr)
pred = (sup.predict_proba(Xte)[:, 1] >= 0.5).astype(int)

tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print('at the default 0.5 threshold')
print(f'  accuracy {accuracy_score(yte, pred):.6f}')
print(f'  caught   {tp} of {int(yte.sum())} frauds')
print(f'  flagged  {tp + fp} transactions')

at the default 0.5 threshold
  accuracy 0.995611
  caught   9 of 236 frauds
  flagged  19 transactions

A model at the default threshold, 99.56% accurate, catching 9 of 236 frauds. Accuracy is not measuring anything anyone wants.

7bComparing candidates

iso = IsolationForest(n_estimators=300, contamination=0.01, random_state=SEED).fit(Xtr)
log = make_pipeline(StandardScaler(),
                    LogisticRegression(max_iter=3000, random_state=SEED)).fit(Xtr, ytr)

scores = {
    'Isolation forest, unsupervised':   -iso.score_samples(Xte),
    'Gradient boosting, supervised':     sup.predict_proba(Xte)[:, 1],
    'Logistic regression, supervised':   log.predict_proba(Xte)[:, 1],
    'Rank by amount':                    Xte['amount'].values,
}
for nm, s in scores.items():
    ap = average_precision_score(yte, s)
    print(f'{nm:34s} AUC {roc_auc_score(yte, s):.4f}  AP {ap:.5f}  '
          f'lift {ap / yte.mean():.2f}')

Isolation forest, unsupervised     AUC 0.7832  AP 0.03370  lift 7.71
Gradient boosting, supervised      AUC 0.8030  AP 0.12566  lift 28.75
Logistic regression, supervised    AUC 0.8027  AP 0.14583  lift 33.37
Rank by amount                     AUC 0.5328  AP 0.00513  lift 1.17

The simplest supervised model wins. Ranking by amount, which is the current process, is barely above the base rate.

7cHyperparameter tuning

None worth reporting, and that is the finding rather than an omission. The lever that matters here is not a hyperparameter, it is the threshold, and section 7d sets it from the review capacity rather than from anything the model produces.

7dFinal evaluation

The queue holds 1,800 cases over the test window, which is 200 a day for two analysts. Everything is measured at that number.

K = 1800   # what the team can review over the test window


def at_k(s, k):
    idx = np.argsort(-s)[:k]
    caught = int(yte[idx].sum())
    return caught, caught / k, caught / yte.sum()


for nm, s in scores.items():
    caught, prec, rec = at_k(s, K)
    print(f'{nm:34s} caught {caught:3d}   precision {prec:.4f}   recall {rec:.4f}   '
          f'lift {prec / yte.mean():.1f}x')

Isolation forest, unsupervised     caught  81   precision 0.0450   recall 0.3432   lift 10.3x
Gradient boosting, supervised      caught  92   precision 0.0511   recall 0.3898   lift 11.7x
Logistic regression, supervised    caught 102   precision 0.0567   recall 0.4322   lift 13.0x
Rank by amount                     caught   9   precision 0.0050   recall 0.0381   lift 1.1x

The same review capacity, filled four ways. The current process finds 9. Labels are worth 21 more than no labels.

What each ranking actually flags

The unsupervised detector flags transactions averaging 388.80 pounds against an overall mean of 71.21. It is doing exactly what it was built to do, which is find the unusual, and section 3c already established that unusual is mostly legitimate. The supervised queue is 67.8% missing-CVV transactions and 28.6% country mismatches, because those are what the labels taught it to look for.

for k in (100, 250, 500, 1000, 2000, 4000):
    line = f'top {k:5,d}  '
    for nm in ['Logistic regression, supervised', 'Isolation forest, unsupervised',
               'Rank by amount']:
        caught, prec, rec = at_k(scores[nm], k)
        line += f'{nm.split(",")[0]:20s} recall {rec:.3f}   '
    print(line)

top   100  logistic recall 0.157   isolation recall 0.030   amount recall 0.000
top   250  logistic recall 0.254   isolation recall 0.051   amount recall 0.009
top   500  logistic recall 0.297   isolation recall 0.123   amount recall 0.017
top 1,000  logistic recall 0.369   isolation recall 0.212   amount recall 0.025
top 2,000  logistic recall 0.445   isolation recall 0.360   amount recall 0.042
top 4,000  logistic recall 0.521   isolation recall 0.445   amount recall 0.106

How the answer changes if the team grows. At every capacity the supervised ranking is ahead, and ranking by amount barely leaves the floor.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not report accuracy. Flagging nothing scores 99.56%.
• Do not use a 0.5 threshold. It catches 9 of 236 here. Set the threshold from the queue length.
• Do not reach for boosting first at this base rate. It lost to logistic regression on the same features, and the reason is that there is very little for it to learn from.
• Do not confuse an anomaly detector with a fraud model. One finds unusual, the other finds fraudulent, and the top one per cent by amount is only 1.78 times more fraudulent than average.
• Do not judge the queue on how many false positives it contains. Judge it on frauds caught per analyst hour.

Reproducibility

What to take from this

• At a base rate under one per cent, every default is wrong. Accuracy, the 0.5 threshold and F1 all optimise for the class you already predict perfectly.
• Set the threshold from the constraint, not from the model. The team can review 200 a day, so that is the queue length and the threshold follows from it.
• Unusual is not the same as fraudulent. Anomaly detection is what you use before you have labels, not what you use instead of them.
• Simple models win on rare events more often than anyone expects. Logistic beat boosting here by 1.16 times on average precision.
• Report precision against the base rate. 5.7% sounds like a broken model and is 12.97 times better than chance.
• The false positives are the asset. They are the only labelled negatives anyone reviewed, and most teams throw them away.

The review team asked for a model that finds fraud. What changed the outcome was not the model, it was refusing to use a threshold of 0.5 and refusing to report accuracy. Both of those are defaults, and on a problem this rare the defaults are the failure mode.