Test type

This sets the condition a variant has to meet before you would ship it.

• Superiority asks whether the variant beats the control at all. Any positive effect that clears your confidence bar counts as a win.
• Non-inferiority asks whether the variant is no worse than the control by more than a margin you accept. Use it when you have another reason to prefer the variant, such as a cheaper implementation or a simpler page, and you only need to prove it does not cost you much. A margin of 2% means you will accept the variant as long as you can rule out a drop larger than 2%.

Number of variants

How many variants ran against the control. Fix this before the test starts. Testing more variants gives more chances for one to look good by luck, which is what the multiple comparisons correction exists to absorb. Dropping variants partway through, or adding them, breaks the maths.

Metric type

• Rate (binomial) is for metrics where each user or session either converts or does not, so the value is a proportion. Enter the count of users and the count of conversions.
• Average (continuous) is for metrics that are an arithmetic mean, such as revenue per user, order value, or pages per session. Enter the count, the mean and the standard deviation, or paste the raw values.

If a user can convert more than once, the metric is an average and not a rate. Average transactions per user is continuous, not binomial.

Raw data format

In raw data mode, paste or upload one column per group in a strict order: the first column is the control, the second is variant A, the third is variant B, and so on. Comma, semicolon and tab separated data all work, and a single header row is detected and skipped. Blank cells are ignored, so columns of different lengths are fine. Export from a spreadsheet as CSV rather than XLSX, which this tool does not read.

Confidence required

The bar a variant has to clear. It is the mirror image of the p-value threshold: 95% confidence means you ship on a p-value at or below 0.05. Raise it when shipping a variant that does not really work would be expensive, and be honest about that cost rather than defaulting to 95% out of habit.

Relative rather than absolute difference

Most calculators test the absolute gap between two rates and then quietly report a relative lift next to it, so the interval you read does not match the test that was run. This tool works on the relative difference throughout. It takes the log of the ratio of the two means, uses the delta method for its standard error, and transforms the result back, so the p-value and the interval both describe the same percentage change you are actually going to quote.

The consequence is that both means have to be above zero. A metric that can be negative, such as profit per user, has no meaningful percentage change through zero.

One-sided testing

The question in an A/B test is normally directional: is the variant better, or better than the margin. A one-sided test answers that question directly and reports the lower bound of the interval, which is the number that matters when you decide whether to ship. Reported confidence levels here are one-sided and are not comparable with the two-sided output of other tools.

Multiple comparisons

With more than one variant against a shared control, the chance of at least one false winner grows with each variant. The correction controls that across the whole test rather than per comparison.

• Dunnett is built for exactly this shape, many variants against one control, and accounts for the fact that the comparisons share the control group. It is the least conservative of the three and the default here.
• Sidak assumes the comparisons are independent, which they are not, so it gives up a little power.
• Bonferroni is the simplest and the most conservative.
• None reports raw p-values. With more than one variant this overstates your confidence, and it is here for comparison rather than for decisions.

Sample ratio mismatch

The tool runs a chi-squared goodness of fit test on the group sizes against an equal split. A very low p-value means the traffic did not divide the way it should have, which usually points at a bug in the assignment, redirect or tracking rather than a real result. When that check fails, fix the plumbing before you read anything else on this page, because the comparison itself is no longer trustworthy. If you deliberately ran an unequal split, ignore this check.

Assumptions and limits

• Normal approximation, so results are approximate with very small groups. Aim for at least a few hundred per group, and treat anything under about 30 as indicative only.
• The correction assumes roughly equal group sizes when modelling how the comparisons relate to each other.
• Session-based metrics break the assumption that observations are independent, since one user contributes many sessions. Intervals will be a little too narrow. Prefer user-based metrics where you can.
• This is a fixed-horizon test. It assumes you fixed the sample size in advance and looked once. Checking repeatedly and stopping when a result crosses the line inflates the false positive rate well beyond the number shown here, and needs a sequential method instead.