The Mathematics Behind the Churn Model

Churn prediction is where machine learning meets a profit-and-loss statement, and it is the problem that most cruelly punishes a model built to maximise accuracy. A subscription business loses perhaps 5 percent of its customers a month, which means a model that predicts “nobody will churn” is right 95 percent of the time and completely worthless. Accuracy, the metric beginners reach for first, rewards exactly the model that does nothing. This is not a quirk of churn; it is the general behaviour of accuracy on any rare event, and it is the first of several places where the naive quantity and the useful quantity point in opposite directions.

The deeper issue is that a churn model is not really a classifier at all. Its job is not to sort customers into “will churn” and “will not”; it is to estimate a probability, so that a business can decide who to spend a retention budget on. That reframing changes everything downstream. It means the model must output honest probabilities, not just good rankings, so calibration matters. It means the decision threshold is an economic quantity set by the cost of a wasted offer against the cost of a lost customer, not a default of 0.5. And it means the popular fix for imbalance, resampling the data, quietly breaks the very probabilities the business depends on. This article works through that chain.

Churn as a probability, not a label

Model the outcome as a Bernoulli variable: customer i either churns ($y_i = 1$) or does not ($y_i = 0$), and the model estimates the probability of churn given what we know about them:

A logistic regression produces this by squashing a linear score through the sigmoid, and any probabilistic classifier produces something analogous:

The output is a number between 0 and 1, and the temptation is to threshold it at 0.5 and call anything above it a churner. Hold that thought, because the entire argument of this article is that 0.5 is almost never the right line, and that the number itself is only useful if it means what it says.

The accuracy paradox, quantified

First, why accuracy fails. Let the churn rate, the base rate, be $\pi$. The “predict nobody churns” model has accuracy:

For $pi = 0.05$ that is 95 percent, and any real model must beat 95 percent merely to look better than doing nothing, even though the null model catches zero churners and saves zero customers. Accuracy is a weighted average of performance on the two classes, and when one class is 95 percent of the data, that class dominates the average and the rare class, the only one you care about, contributes almost nothing.

The lesson is that on imbalanced problems, accuracy measures the base rate more than it measures the model. The metrics that survive imbalance look only at the class of interest, and they come from the confusion matrix.

The confusion matrix as an economic object

Every prediction falls into one of four cells: true positive (predicted churn, did churn), false positive (predicted churn, did not), false negative (predicted no churn, did churn), and true negative. Two ratios read the rare class directly. Precision asks, of the customers we flagged, how many really churned:

Recall asks, of the customers who actually churned, how many did we catch:

These pull in opposite directions, which is the heart of the problem. Flag everyone and recall is perfect but precision collapses, since most of those flagged were not going to churn. Flag only the one customer you are certain about and precision is perfect but recall is near zero. The F1 score summarises the tension as their harmonic mean:

The harmonic mean is used rather than the ordinary average because it punishes imbalance between the two: a model with precision 0.9 and recall 0.1 has an arithmetic mean of 0.5 but an F1 of about 0.18, correctly refusing to reward a model that is good at one measure only by being useless at the other.

But F1 has a hidden assumption that is wrong for churn: it treats precision and recall as equally important. In a real business they are not, because a false positive and a false negative cost different amounts of money. That is where the mathematics has to leave the standard metrics behind and become economic.

Threshold economics: where 0.5 comes from, and why it is wrong

Assign the errors their real costs. Let $C_{FP}$ be the cost of a false positive, a retention offer wasted on someone who was not going to leave, and $C_{FN}$ the cost of a false negative, the lost lifetime value of a churner you failed to catch. For a customer with churn probability p, compare the two actions.

If we flag them and send the offer, the expected cost is the chance the offer was wasted, since they would have stayed anyway:

If we do not flag them, the expected cost is the chance we lose a churner we ignored:

We should flag the customer whenever flagging is the cheaper action:

Solve this for p. Expanding and collecting terms:

which gives the optimal decision threshold:

This small formula is the most important result in applied churn modelling, and it says something the default 0.5 conceals. The right threshold depends only on the ratio of costs, not on the model. When a wasted offer and a lost customer cost the same, $C_{FP} = C_{FN}$ and $t^* = 0.5$, which is the only situation in which the default is correct. But churn almost never has symmetric costs. If losing a customer costs twenty times more than a wasted offer, then:

You should flag anyone with more than a 5 percent chance of churning, not more than 50 percent. A model thresholded at 0.5 would ignore the overwhelming majority of at-risk customers, not because it is a bad model but because 0.5 encodes the false belief that the two errors cost the same. Tuning the threshold to $t^*$ is not fine-tuning; it is the difference between a model that reflects the business and one that fights it.

Why calibration is not optional

The threshold formula has a silent prerequisite that most churn projects miss. It compares the model’s probability p against a cost-derived number $t^*$, and that comparison is only valid if p is a true probability. If the model says 0.05 but such customers actually churn 20 percent of the time, then flagging everyone above 0.048 flags almost nobody who matters, and the carefully derived threshold is applied to numbers that do not mean what they claim.

A model is calibrated when its stated probabilities match reality: among all customers assigned a churn probability of 0.3, close to 30 percent actually churn. Formally:

Calibration is distinct from discrimination, and the distinction matters. A model can rank customers perfectly, always scoring churners above non-churners, and still be badly calibrated, if its probabilities are all compressed toward the middle or shifted toward zero. Ranking is what AUC measures; calibration is what the threshold economics needs. A churn model can have an excellent AUC and produce probabilities that make the cost-optimal threshold meaningless.

The standard way to see calibration is the reliability diagram, which bins customers by predicted probability and plots the observed churn rate in each bin against the predicted one; a calibrated model lies on the diagonal. The standard way to measure it is the Brier score, the mean squared error of the probabilities themselves:

The Brier score is small only when the probabilities are both confident and correct, and it decomposes cleanly into a calibration term and a refinement term, so a model can be penalised specifically for probabilities that do not match observed frequencies. Log loss, the cross-entropy the model is usually trained on, is another proper scoring rule, meaning it is minimised only by honest probabilities and cannot be gamed by a model that lies about its confidence. This is worth holding onto for the next section, because the popular imbalance fix breaks exactly this property.

How resampling breaks the probabilities

Faced with 5 percent churn, a common instinct is to rebalance the training data, oversampling churners or undersampling the rest until the classes are equal. This often improves the raw classification metrics, and it does something dangerous underneath: it changes the base rate the model learns, so the probabilities it outputs are calibrated to the resampled world, not the real one.

The effect is precise and correctable. Training on data where the churn rate has been forced to $\pi_{\text{train}}$ when the true rate is $\pi_{\text{true}}$ shifts the model’s log-odds by the difference in the prior log-odds. The corrected probability is recovered by adjusting the log-odds back:

Read what this does. If you balanced the classes so $\pi_{\text{train}} = 0.5$, the last term is zero, and since $\pi_{\text{true}}$ is small the middle term is a large negative number, pulling every probability sharply downward. A model trained on balanced data that confidently outputs 0.5 for a customer is really saying something much smaller once corrected for the true rarity of churn. Skip the correction and every probability is inflated, the threshold economics operate on fictional numbers, and a retention budget is spent on the wrong people.

This is the mathematical case for a discipline the churn code-along follows: prefer to leave the base rate alone and handle imbalance through the threshold and the metrics rather than by resampling, or, if you must resample, apply the prior correction afterward and recalibrate. Class weights in the loss function are often the cleaner lever, because they change how errors are penalised without falsifying the base rate the probabilities are calibrated to.

Expected value: putting it all together

The threshold formula generalises into a single objective that ties the whole model to money. The expected value of a policy that flags customers above threshold t is the sum over all four confusion-matrix outcomes weighted by their probabilities and their payoffs:

where each v is the value or cost of that outcome. Because the counts in each cell depend on the threshold, this is a function of t, and the business-optimal threshold is the one that maximises it:

This is the honest objective of a churn model, and it reveals how far accuracy sat from the point. Accuracy weights all four cells equally and ignores their costs entirely; expected value weights them by real payoffs and asks the only question the business cares about, which is how much money the policy makes. Sweeping t across its range and plotting expected value turns the model into a decision, and the peak of that curve, not 0.5 and not the point of maximum F1, is where the threshold belongs.

What the mathematics tells you to do

Never judge a churn model by accuracy, because on a 5 percent base rate the do-nothing model already scores 95 percent and catches no one. Read precision and recall instead, and F1 only when you genuinely weight the two errors equally, which for churn you almost never do. Set the threshold from the cost ratio, $t^* = C_{FP}/(C_{FP} + C_{FN})$, because the default 0.5 silently assumes a wasted offer costs as much as a lost customer, and it does not. Check calibration before trusting any probability, since the threshold economics compare the model’s p against a real number and are meaningless if p is not a real probability. Be wary of resampling, because balancing the classes recalibrates the model to a base rate that does not exist, and apply the prior correction if you resample anyway. And where you can, optimise expected value directly, since it is the only objective that measures what the model is actually for.

The idea underneath all of it is that a churn model is a decision system, not a prediction contest. Its output is a probability, that probability feeds a cost calculation, and the cost calculation produces an action. Every link in that chain has a mathematical requirement: the probability must be calibrated, the threshold must reflect the economics, and the metrics must ignore the majority class that accuracy is mesmerised by. Get any link wrong and the model can have a beautiful AUC while losing money, which is the specific and expensive way that churn projects fail. The mathematics is not decoration on top of the business problem here. It is the business problem, written down.

See you soon.