What the MDE is, and what it is not

The minimum detectable effect is the smallest true effect your sample can find reliably, where reliably means at the power level you set. It is a property of the test design, not a prediction about your variant. An MDE of 8% does not mean the variant will lift by 8%; it means that if the truth were an 8% lift, you would catch it about as often as your power setting says, and if the truth were smaller than that you would probably miss it.

Read it as a floor on ambition. If your MDE is 15% and nobody at your company believes this change moves the needle by more than 3%, the test cannot answer the question and running it anyway just buys an inconclusive result at full price.

Relative versus absolute

Both numbers describe the same effect. The relative lift is the percentage change against the baseline; the absolute improvement is the change in percentage points. Going from a 5% rate to 5.4% is a 0.4 percentage point absolute improvement and an 8% relative lift. Quote whichever your audience thinks in, but be explicit about which one you mean, because mixing them up is one of the most common ways A/B test results get overstated.

This tool tests the relative difference, matching the companion significance and sample size calculators, so all three agree with each other.

The power curve

The chart shows power for every effect size, not just the one that hits your target. It is the more honest view: power does not switch on at the MDE, it climbs smoothly. The curve makes the cost of chasing small effects visible, since the left-hand end is flat and near the false positive rate, and it shows how much headroom you have above the MDE.

Number of variants

Two things happen when you add variants. The traffic splits further, so each group gets less, and the bar per comparison rises to control false winners across the whole test. Both push the MDE up. Dunnett is the default because it accounts for the comparisons sharing a control group and costs the least; the others are offered for comparison.

Non-inferiority

For a non-inferiority design the effect is measured against the baseline minus your margin, so the detectable effect can be negative. A result of minus 2% means that with this sample you could show a variant is non-inferior even if it is genuinely 2% worse, because 2% worse is still comfortably inside a wider margin. That is the design working as intended, not an error.

Assumptions

• A fixed sample size decided in advance, with a single evaluation at the end. If you intend to watch the results and stop when they look good, these guarantees do not hold and you need a sequential design instead.
• A one-sided test of the relative difference in proportions, using a normal approximation, on the log of the ratio with a delta-method standard error.
• Equal traffic split across the control and all variants.
• The normal approximation gets rough with very small groups or very rare conversions. The tool warns when your inputs enter that territory.