A colleague hands you a training set and says the churn model gets 0.99 AUC. It does. The score is real and reproducible, and the model is worthless.
This file has 8,000 rows, 2,495 of whom churned, and six deliberate faults. Four are target leakage wearing different clothes. One is not a column at all. Working out which columns you are not allowed to use, and being able to say why for each, is the entire exercise.
If you would rather try it first, the download includes a brief with the task and nothing else.
What you are given
Eleven columns: customer id, signup date, plan, monthly spend, tenure in months, support tickets, days since last login, cancellation reason, days to renewal, account status, and the target, churned. Base rate 31.19%.
Three of those columns are honest, useful fields that a real CRM would have. They are also the ones that will destroy your model.
1. cancellation_reason, the perfect leak
Filled for every one of the 2,495 churners. Empty for all 5,505 who stayed.
filled = df["cancellation_reason"].str.strip() != ""(filled == (df["churned"] == 1)).all() # True
The column is the target spelled differently. A model given it scores 100%.
Notice why it exists. The field is genuinely useful and it is genuinely only populated when someone cancels. That is the pattern worth internalising: leakage usually arrives in a column that is honest, sensible, and recorded after the outcome.
2. days_to_renewal, a leak by arithmetic
Negative for every churner, from minus 90 to minus 1. Positive for everyone else, 1 to 365. It was computed at export time rather than at prediction time.
((df["days_to_renewal"] < 0) == (df["churned"] == 1)).all() # True
The general rule is broader than this column. Any value that depends on when you computed it is suspect. Ask of every feature: would this have existed at the moment I need the prediction, and would it have had this value?
3. account_status, the target under another name
Closed for 2,495, active for 5,505. A perfect duplicate of the label.
Cross-tabulate every categorical column against the target before you model anything. A column with a perfect association is not a strong feature. It is the answer.
4. support_tickets, the subtle one
Churners average 3.37 tickets against 0.84 for those who stayed. It is not a perfect leak, which is exactly what makes it dangerous: it survives the checks that catch the first three, and on its own it scores 0.929 AUC.
Some of that gap is real. Unhappy customers do raise more tickets before they leave. Some is contamination, because the count includes tickets raised after cancellation. From this file alone you cannot separate the two.
The tell is at the edge of the distribution. Not one churner has zero tickets, while 205 retained customers have three or more. A feature that never takes its lowest value for the positive class is describing the outcome rather than predicting it.
The fix is to recompute it with a cut-off, counting only tickets raised before the prediction date. If you cannot get that, drop it and write down why.
5. customer_id encodes time
IDs run 10000 to 17999 in signup order, across dates from January 2024 to January 2026. Correlation with signup date is 1.0000.
ids = df["customer_id"].str.replace("CUST-", "").astype(int)ids.corr(pd.to_datetime(df["signup_date"]).astype("int64")) # 1.0
That matters because churn drifts. It runs 28.5% for customers who signed up in the first quarter of 2024 and 42.9% for the last cohort in the file. An ID left in the feature set is a proxy for the calendar, and a model will happily use it.
6. The split, which is not a column at all
The data is time-ordered, so a random train and test split puts early and late customers on both sides. The model is trained partly on the future and tested partly on the past.
Here is where the usual advice needs care. You will often read that an honest split scores lower. On this file it does not:
With the three clean features, a random split gives 0.658 and a time-ordered split gives 0.660. They are the same number.
Put customer_id in on its own and the mechanism becomes visible. Random split: 0.549. Time-ordered split: 0.501.
An ID is pure calendar and carries no information about a customer. A random split still pays it 0.05 of AUC, because the test set contains the same months the model trained on. An honest split takes that away and the ID falls to exactly a coin flip.
That gap is the whole argument. You split on time not because the number will drop, but because a random split rewards a feature for knowing what month it is, and production will not.
What it actually scores
| Features | Cross-validated AUC |
|---|---|
| days_to_renewal alone | 1.0000 |
| days_to_renewal and support_tickets | 0.9999 |
| clean three plus support_tickets | 0.9412 |
| monthly_spend, tenure_months, last_login_days | 0.6698 |
0.67 is the real answer. It is a usable model. It is not a good story, which is precisely why leakage is so persistent: the leaky version gets praised and shipped, and nobody finds out until it is live.
Look at the third row for a moment. Drop the three perfect leaks, keep only the subtle one, and you still get 0.94. The most dangerous leak in any dataset is never the obvious one. It is the one that leaves you with a number good enough to celebrate and not so good that anyone asks questions.
The habit that catches all four
Before modelling, for every column, answer one question in writing:
At the moment I need this prediction, would this value exist, and would it have this value?
Four of the six faults fail that question outright. The other two are about how you split rather than what you feed in.
A model that scores far better than the problem deserves is not good news. It is a bug report.
Have fun.
