A/B Testing Statistics: A Complete Guide

A/B testing exists to answer a question the everyday data cannot: not what happened, but why. The world is noisy, observation is cheap, and most of what you can passively measure is incapable of supporting a causal claim. This guide builds the statistical foundation that turns a noisy comparison into a trustworthy decision, from why experiments beat observation, through significance, power, and sample size, to the subtler traps of multiple comparisons, concurrent tests, skewed metrics, percentage change, and the right way to frame and communicate a result.

Why Observation Is Not Enough

Observational data is gathered passively, by watching without intervening, and it is almost useless for causal claims because it cannot rule out the alternative explanations for a correlation. A pattern between two metrics can arise from direct causation, from reverse causation, from a confounding variable that drives both, from a feedback loop in both directions, from sheer coincidence, or from a measurement artefact such as a broken tracking pixel that distorts both series at once. The classic cautionary tale is a feature that appeared to halve bounce rate, then a later change that appeared to reverse the gain, when in truth the first release had broken tracking and the second had fixed it. Neither change caused what the numbers seemed to show.

The experiment is the way out, because it builds a concurrent control group rather than comparing a period before a change to a period after it. A before-and-after comparison pins the baseline in the past, so any external event, a season, a campaign, a news story, contaminates only the after period and masquerades as an effect. A randomised A/B test exposes both groups to those same external forces simultaneously, so the only systematic difference between them is the intervention itself. Everything else is distributed randomly between the groups and becomes quantifiable noise rather than hidden bias.

Noise and the Normal Distribution

Even when nothing changes, metrics fluctuate. An A/A test, where both groups receive an identical experience, can still produce a dramatic apparent winner if you read it at the wrong moment, and tools will happily declare such a non-difference significant. This is not a malfunction; it is noise behaving exactly as noise does.

When a metric has a stable mean and variance over time, its values form a bell-shaped normal distribution, which is fully described by just two parameters.

That distributional structure is precisely what makes inference possible: it lets us say how surprising a given observation would be if nothing had really changed. Encoding those assumptions, which probability family the data follows, whether observations are independent, and whether the distribution stays stable over time, is what a statistical model does, and every test rests on one. Independence is the assumption most often quietly violated, since repeat sessions from the same user are not independent of each other.

From Business Question to Hypothesis

A test begins by translating a business question into a statistical one. “Is the new checkout at least as good as the current one?” becomes a substantive hypothesis that the new design does not harm conversion, which becomes a pair of formal hypotheses about the difference between variant B and control A.

The null hypothesis is the default, cautious claim, and the decision rule is to reject it only when the observed data would be sufficiently unlikely if it were true. The whole chain runs from business question to substantive hypothesis to statistical hypothesis, then to an experiment with pre-set parameters, then to inference, and only then back to a business action.

Measuring How Far Is Far Enough

Two tests can show the identical observed rates, say ten percent against twelve percent, yet imply completely different things, because their variability differs. What is needed is a single number that captures the observed gap, the variability of the data, and the sample size all at once. Building it starts with the standard deviation, which measures how spread out the data is around its mean.

The trouble is that standard deviation is expressed in the metric’s own units, so an SD in pounds cannot be compared to an SD in conversion rate. The Z-score solves this by standardising the distance from the model, dividing the gap by the standard error.

For the difference between two proportions, the Z-score uses a pooled estimate of the common rate under the null.

Three forces drive the Z-score: a larger observed difference raises it because there is more signal, a larger sample raises it because there is less noise per observation, and a larger variance lowers it because noise obscures the signal.

The p-value then converts that Z-score into a probability: the chance, if the null were true, of seeing a result as extreme or more extreme than the one observed.

A small p-value means one of three things is true: the null is false and a real effect exists, the null is true but a rare outcome occurred, or the model is misspecified. It is crucial what a p-value is not. It is not the probability that the result is due to chance, nor the probability that the null is true, nor the probability the result will replicate. Under the null, p-values are uniformly distributed between zero and one, which is why running a hundred A/A tests at a five percent threshold will produce roughly five significant results purely by chance.

A few Z-score and confidence-level correspondences are worth committing to memory.

Z-score	One-sided confidence	Two-sided confidence	One-sided p
1.282	90%	80%	0.100
1.645	95%	90%	0.050
1.960	97.5%	95%	0.025
2.326	99%	98%	0.010
2.576	99.5%	99%	0.005

The confidence interval expresses the same information as a range, the set of effect sizes consistent with the data.

There is a clean duality here: any null value lying entirely outside a (1 minus alpha) confidence interval is rejected at level alpha, and as the p-value approaches alpha, the interval bound approaches the null point.

Power and Sample Size

Controlling false positives is only half the job. The other half is avoiding false negatives, and the two errors form a familiar grid. A Type I error, with probability alpha, is declaring a winner when none exists. A Type II error, with probability beta, is missing a true winner. Power, equal to one minus beta, is the probability of detecting a true effect of a given size. The null is deliberately framed so that the Type I error corresponds to the more consequential mistake.

Power depends on the true effect, the sample size, and the threshold.

At a true effect of zero, power equals alpha; as the effect grows, power rises toward one. The effect that matters is the minimum effect of interest, the smallest true improvement worth implementing, which is a business decision rather than a statistical one.

Several levers trade off against each other: more sample raises power, a larger minimum effect is easier to detect, a looser threshold raises power at the cost of more false positives, and more variance lowers power. The only lever that buys power without accepting more risk is sample size, and it is expensive, because the required sample scales with the inverse square of the effect size. Halving the effect you want to detect roughly quadruples the users you need.

Relative improvement to detect	Approximate users per arm (10% baseline, 80% power)
20%	~6,400
10%	~25,000
5%	~100,000
2%	~625,000
1%	~2,500,000

A test that cannot detect a business-meaningful effect is underpowered, and a programme of underpowered tests looks fruitless even when the ideas are good; analyses of public A/B tests have found a large majority to be underpowered. A test that runs far longer than needed to detect effects too small to matter is overpowered, paying in delay for no extra protection. A properly powered test rests on a minimum effect that is feasible, meaningful, and achievable within a reasonable timeframe, with eighty percent power common and ninety percent preferable for high-stakes calls. Critically, power must be set before the test; computing it afterward from the observed effect is circular and misleading.

Multiple Comparisons

Comparing one control against several variants introduces a new hazard. If you declare a winner whenever any one comparison is significant, the chance of at least one false positive across the family of tests exceeds alpha.

With three variants at a five percent threshold, the family-wise error rate is about fourteen percent, nearly triple the intended rate. Presenting only the best of several variants as if it were the only one tested is like firing several arrows, discarding the misses, and claiming to be a marksman.

The simplest correction divides the threshold by the number of comparisons.

This Bonferroni approach is easy but conservative, because it ignores that every variant is compared against the same shared control, which correlates the tests. A step-down procedure that exploits that shared-control correlation (Dunnett’s method) is more powerful for the same protection. Either way, adding variants inflates the sample needed or, at fixed sample, raises the minimum detectable effect, so each extra variant makes the test less sensitive. The lesson behind the famous test of dozens of shades of blue is exactly this: testing many variants at once is hugely inefficient, and a smarter path is to test a few spread-out options, then run a follow-up near the winner.

Concurrent Tests

Running several tests at once is efficient, and it is surrounded by a persistent myth: that concurrency inflates false positives. It does not. The p-value calculation already absorbs whatever variance is present, wherever it comes from, so if one test adds variance to another, that test’s standard error grows and its p-value reflects it. The Type I error rate is preserved; what suffers is power, since more variance means less sensitivity.

The real risk is external validity, whether a result obtained during the test still holds after deployment. A variant might win only because another test’s variant was running at the same time, and once that other variant is gone, the winner may no longer perform. This is not a false positive; it is a failure to generalise. The tempting fixes each have costs: running tests sequentially eliminates interaction but collapses throughput, and isolating users into separate lanes hides a worse problem, that the eventually-deployed combination of two winning variants may never have been experienced by any user. The practical recommendation is to run in parallel and actively monitor for interactions by segmenting one test’s results by the variant of the other; if the segments diverge sharply, an interaction is plausible and one test can be paused and rerun.

When the Metric Is Not a Proportion

Many of the most important metrics, revenue per user, order value, sessions per user, are not proportions. They are skewed, full of zeros, and look nothing like a bell curve in raw form, which provokes the worry that Z-tests and t-tests no longer apply. They do, because these tests compare means, not individual values, and by the central limit theorem the sampling distribution of a mean approaches normality as the sample grows, whatever the shape of the underlying data.

So it is never the raw revenue that needs to be normal, only the sampling distribution of its mean, which the central limit theorem delivers for samples beyond a few dozen to a few hundred observations. There is no need for rank-based or bootstrap alternatives unless you are testing something other than the mean, such as the median. The one genuine extra step is the standard deviation. For a proportion it follows directly from the rate.

For a continuous metric no such shortcut exists, so the standard deviation must be computed from the raw data, including the zeros for non-converters, which is exactly the figure that cannot be recovered from a tool that only reports an average. Because revenue per user combines conversion rate and order value, its variance is higher than either component’s, so switching a test from conversion rate to revenue per user typically raises the required sample by twenty to forty percent.

The Percentage-Change Trap

Almost every result is communicated as a percentage lift, yet almost every tool computes p-values and confidence intervals for the absolute difference, and naively combining the two systematically understates uncertainty. The standard error for an absolute difference is not the standard error for a relative one, because dividing by the baseline changes the variance.

If you compute a p-value for the absolute difference and then report the finding as a percentage lift, you have used the wrong standard error: the p-value comes out too small and the interval too narrow. The bias runs in one direction only, always understating uncertainty, and it grows with the size of the true effect. The correct interval for a relative change uses the coefficient of variation in place of the standard deviation.

There is no closed form for the corresponding p-value, so it is found iteratively by adjusting the Z-score until the lower confidence bound just touches zero. For planning, multiplying the absolute-difference sample size by about 1.04 keeps a relative-change test on the safe side.

Asking the Right Question

Not every test is a simple “is B better than A,” and the framing changes both the null and the efficiency. The long-running debate over one-sided versus two-sided tests usually misses that the two answer different questions. A one-sided test asks whether the variant is better; a two-sided test asks whether the two are exactly equal, a null that rarely matches any real business decision. The one-sided test, putting its whole rejection region in one tail, reaches significance with substantially fewer users for the same confidence, which is not cheating but using the right test for the actual question.

Four hypothesis shapes cover most real situations. Simple superiority, where any positive lift justifies adoption, suits cheap and reversible changes.

Strong superiority raises the bar by a margin, fitting changes that carry ongoing cost.

Non-inferiority lowers the bar, allowing a variant that is slightly worse on the primary metric to pass when external factors, lower cost or strategic value, make it preferable anyway.

The point-null two-sided test, asking whether two things are exactly equal, is genuinely rare in this setting. On identical data, these framings yield different significance levels, because only the question has changed, not the numbers, which is why the hypothesis must be chosen before the data is seen, never after.

When a test carries more than one outcome metric, the multiple-comparison problem returns. The fix is to name one primary metric, as close to the bottom line as possible, that decides the test, and to keep secondary metrics for diagnostic context. Avoid co-primary metrics that are mathematically nested in each other, such as average order value alongside revenue per user, or add-to-cart rate alongside purchase rate. Whether a correction is needed depends on the decision rule: requiring all primary metrics to be significant is conservative on its own and needs no correction, whereas adopting the variant if any one is significant opens a false-positive pathway per metric and does. Step-down procedures such as Holm-Bonferroni or the slightly more powerful Šidák handle this.

Communicating the Result

A perfectly run test still fails if its result is communicated badly, and the antidote is a three-stage process. First, prepare stakeholders by building intuition for variability: run A/A tests on their own data and show how often pure noise would have triggered a “winner,” and run many to show that p-values under the null are uniformly spread, so a few will always look significant. The aim is to move people from “p equals 0.001, so this is almost certainly real” to “statistical tests limit the rate of wrong decisions, not the correctness of any single one.”

Second, pre-define everything before the test starts: the primary metric and its definition, the significance threshold, the minimum detectable effect and duration, the error rates, and the action to take for each possible outcome. If the team cannot agree what it will do for each outcome, the test should not run, because either the decision is already made or the question is not well formed.

Third, present results honestly. Good graphs are self-contained, label their metrics in full, and always show uncertainty as an interval rather than a bare point estimate. Tables should carry the test’s metadata, state the null hypothesis beside any p-value, and round consistently. A negative result deserves particular care: a non-significant test does not show the variant has no effect, only that the evidence was insufficient at the agreed threshold, and the honest framing is that acting on a p-value of, say, 0.12 would mean making mistakes at more than double the rate you accepted when you designed the test. Above all, never revise the threshold after seeing the data; the threshold exists precisely to prevent that reasoning, and if it needs changing, it changes for future tests, never the one in hand.

Planning and Reading a Test

Planning a standard test runs through a short checklist: set a hard deadline, choose the hypothesis type, fix the number of variants and the metric type, pull the eligible users per week and the baseline rate from recent history restricted to the exact segment that will be exposed, set the confidence level, and choose a duration off the power curve. That curve, plotting the minimum detectable effect against duration at a fixed power, shows sharply diminishing returns, so the right duration is the point where the detectable effect is both achievable for the kind of change being tested and meaningful for the business. Continuous metrics need one extra input that proportions do not, the baseline standard deviation, and if the real variance during the test exceeds the planned figure, power falls and the test may need extending.

Reading a completed test means reporting the outcome, the users per arm, the duration, the confidence level, the relative lift, and crucially the confidence interval on that lift, plus a sample ratio mismatch check that confirms the allocation was actually random. A significant result’s estimated lift is the single most probable true effect, not a guarantee, and the interval bounds the plausible range. Checking the sample ratio first matters because an allocation bug can invalidate every conclusion that follows.

The Through-Line

Underneath all of it sits one discipline: decide the question, the threshold, and the action before the data arrives, then read the data through a model that honestly accounts for noise, variance, and sample size. Experiments beat observation because randomisation turns hidden bias into measurable noise. The Z-score, p-value, and confidence interval are three views of the same distance from the null. Power and sample size are the cost of sensitivity, and that cost grows steeply as the effect you chase shrinks. And every refinement, multiple-comparison corrections, the percentage-change standard error, the choice of one-sided superiority or non-inferiority, exists to keep the rate of wrong decisions at the level you agreed to accept. Get that framework right and an A/B test stops being a coin you read hopefully and becomes a reliable instrument for learning what actually works.