Why fix the sample size in advance

A fixed-horizon significance test only means what it claims if the sample size was decided before the test started and the data was looked at once, at the end. Peeking at the results and stopping when they look good inflates the false positive rate far beyond the stated threshold. This calculator sets that size in advance so the p-values and intervals you read at the end keep their meaning.

The number it controls is the false negative rate: the chance of running an honest test and still missing a real effect you cared about. Power is the flip side of that risk.

Test type

• Superiority plans for showing the variant beats the control at all.
• Non-inferiority plans for showing the variant is not worse than the control by more than the margin. The effect being detected is measured against the baseline minus the margin, matching how the significance calculator will read the result, so the two tools agree.

Number of variants

More variants against one control means more chances for a false winner, so the confidence bar per comparison has to rise, and with it the sample size. The adjustment here mirrors the significance calculator: Dunnett by default, which accounts for the comparisons sharing a control and costs the least sample. This number cannot change mid-test.

Minimum effect of interest

The smallest relative lift worth detecting, as a percentage of the baseline. Going from a 0.4% conversion rate to 0.5% is a 25% relative lift. Pick the smallest lift that would still justify having run the test, not the lift you hope for. Halving the minimum effect roughly quadruples the required sample, so this input dominates everything else.

Baseline values

For a rate metric the baseline conversion rate is enough, because a proportion’s variability follows from the rate itself. For an average you also need the standard deviation, which is where most of the sample size comes from: high-variance metrics like revenue per user need far more traffic than conversion rates to detect the same relative lift. If you have historical raw data, paste it in raw mode and the tool computes the mean and standard deviation for you.

Confidence required

The same threshold the variant will have to clear when you read the test. It is one minus the p-value threshold. The costlier a false winner, the higher this should be.

Power level

How likely the test is to detect the minimum effect of interest if it truly exists, at the chosen confidence. At 80% power, a real effect of exactly the minimum size is found four times out of five, and missed once. If missing a winner is expensive, plan at 90% rather than 80% and accept the larger sample.

Method and assumptions

• One-sided test on the relative difference, using the log of the ratio of means and the delta method for its standard error, identical to the companion significance calculator.
• The variant under the alternative is assumed to sit at baseline times one plus the minimum effect; for averages, the standard deviation is assumed unchanged from the baseline.
• Groups are assumed equally sized, which is the most sample-efficient split and what the multiplicity adjustment models.
• Normal approximation throughout, so treat results for very small samples or very rare conversions as indicative.
• For heavily skewed continuous metrics, such as revenue per user where most users spend nothing, simulation shows the estimate runs a few percentage points short of target power; the tool suggests a 10% cushion for that case.
• This plans a fixed-horizon test with one look at the data. If you intend to peek and stop early, you need a sequential design, and the numbers here do not apply.

Duration

If you enter your weekly eligible traffic, the tool divides the total required sample by it to estimate the test length. Run whole weeks, and at least two of them, so day-of-week patterns average out. If the estimate comes out at several months, that is the calculator telling you the minimum effect is too ambitious for your traffic, not a number to negotiate with.