A free Excel workbook for reading test results honestly, with a validity gate, corrected significance and a revenue range instead of a revenue guess.
The moment a test ends, everyone in the room asks the same question: did it win? It is the wrong question to ask first, and asking it first is how teams end up shipping changes that do nothing and killing ideas that would have worked.
Check the split before you check the result
Before you look at any conversion rate, compare the traffic split you actually got against the one you asked for. If you planned 50/50 and got 50.8/49.2 across 100,000 users, that is not rounding. That is a sample ratio mismatch, and it means something went wrong in delivery or tracking.
This matters more than it sounds, because the usual causes are not neutral. A redirect that fires slowly loses the impatient users from one arm. A bot filter that catches one variant and not the other strips out a whole class of traffic. Tracking that fails on the variant loses exactly the people who had the worst experience. None of these add noise evenly. They all bias the result in a particular direction, which is far more dangerous than random error, because the number still looks perfectly plausible.
A failed ratio check does not mean your test was inconclusive. It means the data is not evidence of anything, and the honest move is to fix the cause and run it again.
“Not significant” means two different things
This is the part that costs teams the most, and it is entirely fixable.
Imagine two tests. Both come back non-significant. The first has a confidence interval of -0.4% to +0.6%. The second has an interval of -12% to +19%.
Those results are treated identically in most readouts. They are close to opposites.
The first test genuinely answered the question. The effect, whatever it is, is tiny. You have learned that this idea does not move the metric, and you should stop spending time on it. That is a real, valuable finding.
The second test learned nothing at all. The true effect could be a serious loss or a substantial win, and you have no idea which. Calling that “no difference” and moving on throws away the test entirely. The correct response is to run it again with more traffic or a bolder change.
The p-value cannot tell these apart, because it collapses both into the same word. The confidence interval separates them instantly. This is why the workbook puts the interval first and treats “flat” and “inconclusive” as two different outcomes rather than one.
Every extra metric is another chance to fool yourself
Look at eight metrics at 95% confidence and you should expect roughly one of them to come back “significant” by pure chance. Look at three variants instead of one and the same thing happens.
This is not a subtle statistical footnote. It is the single most common way a losing test gets shipped: the primary metric was flat, so someone goes looking, finds a secondary metric that hit significance, and builds the case around that instead.
The fix is unglamorous. If you are making several comparisons, divide your alpha by the number of comparisons. The workbook does this automatically as soon as you add a second variant, so with two challengers your threshold tightens from 0.05 to 0.025 without you having to remember. It is deliberately conservative, which is the point. A result that survives it is real.
Give finance a range, not a number
Revenue per visitor is what the business actually cares about, and it is also the noisiest thing you measure. A test can show a clearly significant conversion win and still be genuinely uncertain about revenue, because order values vary far more than conversion does.
The temptation is to take the observed lift, multiply by annual traffic, and put one confident number in the deck. Do that a few times, watch the numbers fail to appear in the accounts, and your team’s forecasts stop being believed.
Project the range instead. On the worked example in the workbook, a +10% conversion lift with an interval of +2.0% to +18.7% turns into an annual impact somewhere between £97,000 and £913,000. That is an uncomfortable range to present. It is also the truth, and presenting it honestly is what makes the estimate credible when it lands inside it.
The workbook also flags whether average order value moved, because a variant can appear to win on revenue simply by shifting the mix of what people buy rather than converting more of them.
Four outcomes, not two
Ship and do not ship are not enough. A useful readout distinguishes:
- Ship. Significant, positive, and the guardrails are clear.
- Do not ship. Significant and negative.
- Flat. A tight interval around zero. The idea does nothing. Stop iterating on it and spend the traffic elsewhere.
- Inconclusive. A wide interval. The test could not answer the question. Re-run it with more traffic or a bigger swing.
Most testing programmes have no way to record the third outcome, so flat results get quietly relabelled as inconclusive and the same idea comes back three months later in a new costume.
What the workbook deliberately leaves out
There is no observed power calculation. Power worked out after the fact, from the effect you happened to see, is just the p-value wearing a disguise. It will always tell you a null result was underpowered, because that is arithmetically what it does. It feels rigorous and it tells you nothing.
The confidence interval and the minimum detectable effect you planned for do that job properly, and both are already on the sheet.
