The Mathematics Behind A/B Testing

An A/B test looks like the simplest thing in analytics. Show version A to half your visitors and version B to the other half, count the conversions, and declare the higher number the winner. Almost everyone who runs tests this way is fooled regularly, because the one number they are comparing is not a fact about their site. It is a random draw, and two random draws differ even when nothing underneath them does. Flip a fair coin a hundred times against another fair coin a hundred times and one will “win”. The entire discipline of statistical testing exists to answer a single question that the raw comparison cannot: is this difference real, or is it the kind of difference two identical things produce by chance?

This article works through the mathematics that answers it. It explains why a conversion rate has a built-in margin of error, what a p-value actually claims and what it does not, where the sample size formula comes from and why it is the most useful equation in experimentation, and why checking your test every morning quietly destroys its validity. The tone is lighter than the machine learning articles in this series, because the statistics of a two-group test are genuinely simpler, but the one formula worth memorising is derived in full.

A conversion rate is a random variable

Start with what a conversion rate is. Each visitor either converts or does not, which is a Bernoulli trial with some true underlying probability p. Observing n visitors and counting conversions, the estimate is the sample proportion:

This number is not the true rate p; it is an estimate of it, and like any estimate it has a spread. The variance of a single Bernoulli trial is $p(1-p)$, and averaging n independent trials divides that by n, so the sample proportion has:

The standard error is the crucial object. It says how much your measured conversion rate would wobble if you ran the same test again on a fresh sample of the same size. A rate of 5 percent measured on 200 visitors has a standard error of about 1.5 percentage points, so the “5 percent” is really “somewhere around 3.5 to 6.5 percent”. Two variants showing 5.0 and 5.4 percent on such samples have not been distinguished at all; the gap is smaller than the noise in either number. Notice the $\sqrt{n}$ in the denominator: to halve the uncertainty you must quadruple the sample. Precision is expensive, and it gets more expensive the more of it you want.

The two hypotheses, and the courtroom

A test is a decision between two statements about the world. The null hypothesis says the variants are identical; the alternative says they differ:

The logic is deliberately asymmetric, and the courtroom analogy is exact. The null hypothesis is the presumption of innocence: the new variant is assumed to do nothing until the evidence is strong enough to conclude otherwise. We never prove the null true, exactly as a court never declares a defendant innocent, only “not guilty”. We either gather enough evidence to reject it or we fail to. This asymmetry is the source of a great deal of confusion, and it is why “the test was not significant” means “we did not find enough evidence”, not “the variants are the same”.

Two ways to be wrong

Because the decision is binary and the truth is binary, there are two distinct errors, and they are not symmetric in consequence.

A Type I error is convicting the innocent: declaring a winner when the variants are actually identical. Its probability is the significance level, chosen in advance:

The convention $\alpha = 0.05$ means you accept a 1-in-20 chance of crying wolf on any given test. A Type II error is acquitting the guilty: failing to detect a real difference that exists. Its probability is:

and the complement of this is the single most important quantity in test design, the power:

Power is the probability that a real effect, if it exists, will actually be caught. The industry convention is 80 percent, meaning you accept a 1-in-5 chance of missing a genuine improvement. The two error rates trade against each other, and the reason underpowered tests are the silent epidemic of experimentation is that everyone fixes $\alpha$ at 0.05 and forgets to check $\beta$, running tests that had almost no chance of detecting the effect they were looking for.

The test statistic

To decide, we reduce the two samples to a single number measuring how far apart they are in units of noise. The observed difference is $\hat{p}_B – \hat{p}_A$, and its standard error, since the two groups are independent and variances of independent quantities add, is:

The test statistic divides the difference by its standard error:

Read this as the signal-to-noise ratio of the experiment. The numerator is the effect you saw; the denominator is how much effect you would expect from chance alone. A z of 0.5 means the observed gap is half of a typical chance fluctuation, which is nothing. A z of 3 means the gap is three times larger than chance usually produces, which is hard to explain away. The Central Limit Theorem is what makes this work: for reasonable sample sizes the sample proportions are approximately normal, so under the null hypothesis z follows a standard normal distribution, and we know exactly how often a standard normal exceeds any given value.

The p-value, and what it is not

The p-value converts the test statistic into a probability. It is the chance of seeing a difference at least as large as the one you observed, if the null hypothesis were true:

A small p-value means your data would be surprising in a world where the variants are identical, so that world becomes hard to believe. If it falls below $\alpha$, you reject the null and call the result significant.

The p-value is the most misread number in statistics, and two misreadings cause real damage. It is not the probability that the null hypothesis is true; the null is either true or not, and the p-value is a probability about the data assuming the null, not about the null assuming the data. And it is not the probability that your result is a fluke in the sense people mean. A p-value of 0.05 does not mean there is a 95 percent chance your variant is better. It means that if the variant did nothing, data this extreme would occur 5 percent of the time. Keeping the conditional direction straight, probability of data given hypothesis, not hypothesis given data, is the whole discipline of reading a test honestly.

The confidence interval says more

A p-value gives a yes or no. A confidence interval gives the same information plus the effect size, which is usually what you actually care about. The 95 percent interval for the true difference is:

where $z_{\alpha/2} = 1.96$ for 95 percent confidence. The connection to the test is exact rather than approximate: the result is significant at level $\alpha$ precisely when the confidence interval excludes zero. If zero is a plausible value for the difference, you cannot rule out “no effect”.

The interval is more useful than the p-value because it distinguishes cases the p-value blurs together. A significant result whose interval runs from +0.1 to +8 percent tells you the effect is positive but its size is almost unknown. A significant result whose interval runs from +2.0 to +2.4 percent tells you the effect is both real and precisely a bit over 2 percent. Both have the same p-value story, “significant”, and completely different business meanings. Report the interval, and a non-significant result also becomes informative: a tight interval around zero says “genuinely no effect”, while a wide one says “we learned nothing, the test was too small”.

Minimum detectable effect and the sample size formula

Now the practical payoff, the equation that should be run before every test rather than after. Everything above assumes you already have data. Sample size planning runs the logic backwards: given the smallest effect worth caring about, how many visitors must you collect to have a good chance of detecting it?

Define the minimum detectable effect, the smallest true difference you want the test to catch:

The derivation balances the two error rates on the number line. Under the null, the difference is centred at zero with standard error $\text{SE}0$, and we reject when the difference exceeds the critical value $z{\alpha/2},\text{SE}_0$. Under the alternative, the difference is centred at $\delta$ with standard error $\text{SE}1$, and for the test to have power $1-\beta$, that alternative distribution must place $1-\beta$ of its mass beyond the rejection threshold, which requires its centre to sit $z\beta$ standard errors above the threshold. Writing that balance down:

The two z-values are the price of each guarantee: $z_{\alpha/2}$ buys protection against false positives, $z_\beta$ buys power. Approximating both standard errors with a common rate p, so that $\text{SE} \approx \sqrt{2p(1-p)/n}$ per group, and solving for n gives the formula worth memorising:

Every term earns its place. The effect $\delta$ is squared in the denominator, so halving the effect you want to detect quadruples the sample required, which is why small improvements are so expensive to prove. The variance $p(1-p)$ in the numerator means rare events, where p is small, need larger samples, since there is less signal to work with. And the bracket $(z_{\alpha/2} + z_\beta)^2$ shows the cost of your two guarantees: demanding 99 percent confidence or 95 percent power inflates it directly.

Put numbers in it. To detect a lift from 5 percent to 6 percent, so $\delta = 0.01$ and $p \approx 0.055$, at the standard $\alpha = 0.05$ and 80 percent power, the z-values are 1.96 and 0.84:

Over sixteen thousand visitors, for a one-point lift. Run that test on two thousand visitors, as people constantly do, and its power is so low that a real one-point improvement would usually go undetected. The formula’s blunt lesson is that most tests are too small, and you can only know that in advance.

The peeking problem

Here is the mistake that invalidates more tests than all the others together, and it is invisible to the people making it. You plan for sixteen thousand visitors, but you check the dashboard every morning, and one day it shows significance, so you stop and declare victory. That significance is very likely fake.

The reason is that the 5 percent false-positive rate is a guarantee about one single look at the data, taken at the planned sample size. Every additional look is another opportunity for the wandering random difference to cross the threshold by chance. The test statistic drifts up and down as data accumulates, and if you watch it continuously and stop the instant it crosses the line, you are not asking “is the effect real” but “did the noise ever touch the line”, and given enough looks, it always will.

The inflation is severe. A test peeked at repeatedly can have a true Type I error rate not of 5 percent but of 20 to 30 percent, depending on how often you look. Roughly, the false-positive rate grows with the number of looks:

for k independent looks. The fix is one of two disciplines. Either fix the sample size in advance and look exactly once, when you reach it, which is the classic approach the formula above supports. Or adopt a method built for continuous monitoring, sequential testing or the Bayesian approaches some platforms now offer, which spend the error budget deliberately across many looks by raising the bar at each one. What you cannot do is use a fixed-horizon test and peek at it like a sequential one, which is exactly what most dashboards invite you to do.

Testing many things at once

The same multiplicity problem appears in space rather than time. Test five variants against a control, or twenty metrics on one test, and each comparison carries its own 5 percent chance of a false positive. The chance that at least one of them fires by accident is not 5 percent:

For twenty independent comparisons at $\alpha = 0.05$:

Nearly two in three chance that something will look significant when nothing is. This is the same $\sqrt{2\ln M}$ selection effect that inflates tuned model scores, wearing marketing clothes: the more comparisons you run, the more likely one wins on luck alone. The standard corrections tighten the threshold to compensate, the simplest being Bonferroni, which divides the significance level by the number of tests:

so twenty comparisons must each clear 0.0025 rather than 0.05. It is conservative, but it restores the overall false-positive rate to something near the 5 percent you thought you had. The practical rule is to decide your primary metric before the test, and treat everything else as exploratory rather than as twenty separate chances to declare a win.

What the mathematics tells you to do

Compute the sample size before the test, always, because $n = (z_{\alpha/2}+z_\beta)^2 , 2p(1-p)/\delta^2$ tells you whether the test you are about to run can possibly detect the effect you care about, and most cannot. Decide the minimum detectable effect first, since it is squared in that formula and a small target is a large bill. Fix the horizon and look once, because the 5 percent guarantee dissolves the moment you peek, and a dashboard checked daily is running at a hidden false-positive rate several times higher than you believe. Read the confidence interval, not just the p-value, because the interval carries the effect size and tells a tight “no effect” apart from a useless “too small to say”. Name your primary metric in advance, because twenty secondary metrics are twenty chances for noise to win. And keep the p-value’s meaning straight: it is the probability of your data under the null, not the probability that your variant is better.

The single idea beneath all of it is that a conversion rate is a measurement with a margin of error, and A/B testing is the arithmetic of telling a real difference from the wobble in that measurement. The standard error sizes the wobble, the test statistic compares the effect against it, the sample size formula tells you how much data the comparison needs to be trustworthy, and the peeking and multiplicity problems are two versions of the same trap: give randomness enough chances and it will eventually hand you a result you wanted to see. Experimentation done well is mostly the discipline of not giving it those chances.

See you soon.