Most A/B tests fail long before the results come in. They fail at the point somebody says “let’s try the button in green and see what happens.”
That sentence contains no claim. There is nothing in it that could turn out to be wrong. So whatever the test returns, you learn nothing you can carry into the next one. A win you cannot explain will not repeat, and a loss tells you only that this particular green button did not work on this particular day.
A hypothesis fixes that, and it takes one sentence.
The formula
Every testable hypothesis fits the same shape:
Changing [the element] from [current state] to [new state] will increase or decrease [the metric] because [the insight].
Four parts. If any one of them is missing, you are not testing a belief. You are changing something and watching.
The element
The exact part of the page, app or email you plan to modify. Exact enough that somebody else could build it without coming back to ask you a question.
“The checkout page” is not an element. “The button” is not an element. “The primary call to action on step two of checkout” is an element, because there is only one thing it could mean.
The metric
The one number that decides the test, chosen before you look at any data. Conversion rate, click-through rate, completion rate: whichever it is, pick one.
Two primary metrics means two chances to declare a win, and roughly twice the chance of a false one. Name your guardrails separately: the numbers you are not willing to damage in order to move the primary. Refund rate, support contacts, unsubscribes. They do not decide the test. They can stop it.
The insight
The reason you expect the change to work. This is the part the test actually examines. The design change is only how you put the question.
Insight comes from somewhere you can point at: session recordings, support tickets, exit surveys, funnel drop-off, or a test you already ran. “Because it will look better” is not an insight. It cannot be wrong, so nothing can be learned from testing it.
What it looks like assembled
Changing the primary CTA on the checkout page from “Continue” to “Complete order securely” will increase checkout completion rate because users hesitate when nothing tells them whether the next click will charge their card.
Compare that with “let’s make the checkout button green and see if sales go up.”
The first one names the element precisely, records both states, commits to a single metric in advance, and makes a claim about people that could turn out to be false. If it loses, you have learned something real about your users: reassurance at the payment step was not what was holding them back. That finding survives the test and shapes the next one.
The second one leaves you exactly where you started.
Six things to settle before you launch
- The element is named precisely enough for someone else to build it
- Current state and new state are both written down
- One primary metric, and it is the one the business actually cares about
- Guardrail metrics named, with the damage you will not accept
- The insight is a claim about users that could turn out false
- Sample size and run length agreed before a single visitor is exposed
If you cannot fill in all six, the test is not ready. That is a useful finding on its own, and it costs you nothing compared to running a test you cannot interpret.
The template
I have put the whole thing into a PowerPoint template you can fill in and share with whoever signs off on your tests. It has the formula with blanks, a page each for the element, the metric and the insight, a results table to complete after the test, and the checklist above.
There is also a completed example deck showing the checkout hypothesis above worked through from start to finish, including the result.
Once your hypothesis is written, the next question is how many visitors you need before the result means anything. The Sample Size Calculator answers that, the Power and MDE Calculator tells you what a fixed sample can realistically detect, and the Statistical Significance Calculator handles the maths when the test finishes.
Hope this helps you.
