Conversion optimisation is compound interest for a business. Improve your conversion rate by one percent a year and you get a gentle upward curve. Improve it by fifteen percent a year and you get a completely different trajectory, because small consistent gains compound into outsized long-term results. A/B testing is what makes those gains real rather than imagined, and getting the foundations right is what separates a program that compounds from one that just generates noise. This guide covers those foundations end to end: what testing actually is, what to test, how to prioritise, the statistics you cannot skip, and the strategy that ties it together.
What A/B testing is, and what it is not
The first thing to get straight is that A/B testing is not conversion optimisation. Conversion optimisation is the whole process of improving results. A/B testing is the measurement methodology you use to validate whether a change produces real business impact. You can optimise without testing, but then you are guessing what works, and testing simply removes the guessing. So at its core, A/B testing is three things: a measurement methodology, a tool for validating business impact, and a way to manage the risk of change.
It is just as important to know what it is not. It is not a prescription for what to test, that comes from research. It is not a tool for understanding why something worked, only that it worked. And it is not riskier than doing nothing, because change is inevitable whether you test or not, and testing is what makes the necessary change safer. A handful of common myths fall away once you internalise this. Testing is not just for spammy sites, since Google and Facebook run more tests than anyone. Apple does not simply have great vision, it has a CRO team running continuous tests. And testing does not create dark patterns, because testing measures what you choose to build, and what you choose to build is a separate ethical decision.
There is one trap worth flagging up front, because it catches even experienced practitioners. After a test wins, it is tempting to construct an elaborate story about why it won. Resist it. The post-test explanation is still a hypothesis. You validated that something worked, not why, and labelling the story as fact is how programs fool themselves.
What to test
Testing is a way to solve problems, and the precise question it answers is narrow: does this proposed solution, aimed at a known problem, produce measurable business impact? That framing implies a ratio most teams get backwards. Roughly eighty percent of your effort belongs in research, understanding where the problems are, what they are, and why they exist, and only twenty percent in the experimentation itself. The reason is brutal symmetry. Botch the experimentation phase and all your research is wasted, but botch the research phase and you are running flawless tests in the wrong place on the wrong problem.
So the work proceeds problem-first. Start with where the problem is, using analytics to find where traffic drops through the funnel, and concentrate testing there rather than on a low-traffic page like the privacy policy. Then establish what the problem is, using qualitative research, surveys, interviews, polls, and user testing, to find the root cause, whether it is clarity, friction, trust, or relevance. Only then move to what the solution is, with a cross-functional team generating many hypotheses rather than stopping at the first idea, because the tenth and eleventh ideas are often the best. Finally you test, and the A/B test measures the business impact of that hypothesis. There is a second legitimate reason to test beyond solving a known problem, which is shipping a new feature, because new functionality can quietly hurt the business, and a test tells you whether it adds, subtracts, or is neutral before it is too late.
Prioritising what to test
Traffic is always the constraint. You cannot run everything at once, and without a system the loudest person in the room decides what gets tested, regardless of whether their idea has any evidence behind it. A prioritisation framework does two jobs: it forces objectivity, or at least structured subjectivity, and it builds a culture where data-backed ideas beat gut-feel ones.
Several frameworks exist. PIE scores potential, importance, and ease from one to ten, which is quick but highly subjective, and potential is genuinely unknowable before you test, which is the whole reason you are testing. ICE simplifies to high-or-low scoring on impact, cost, and effort, which is clearer but clusters too many ideas at the top. A severity star rating from one to five works well for issues already mapped from research but still leans on subjective judgment. The strongest option is PXL, which replaces subjective scores with binary yes-or-no factual questions: is the change above the fold, is it noticeable within five seconds, does it add or remove an element rather than just tweak one, is it designed to increase motivation, is it on a high-traffic page, was it found through user testing, through other qualitative research, confirmed in analytics, confirmed in heatmaps, and is it under four hours of developer time. Two of those questions score double, because a change nobody can perceive cannot have impact, and bigger structural changes tend to produce bigger effects. Motivation is weighted above friction because it is the more powerful lever on behaviour. The cultural payoff is the real point: an idea with no research behind it scores a cluster of zeros, so teams quickly learn to bring data or watch their ideas sink in the queue. The honest bottom line is that using any framework matters far more than which one, but if forced to pick, choose PXL.
The statistics you cannot skip
Statistics has to answer two questions. Before the test, do we have enough traffic and conversions to run this at all, and after it, is the test done so we can call a winner.
The first is a sample size question, and you answer it with a pre-test calculator rather than a guess, feeding in weekly traffic, weekly conversions, and the number of variants. The calculator tells you how long the test must run and what minimum detectable effect you can realistically catch in that time. That minimum detectable effect, or MDE, is the smallest relative improvement your test can reliably see given your traffic. On a small site, one week might only detect a sixty-five percent uplift, while four weeks brings that down to around twenty-four percent. The practical rule is that four weeks is your maximum duration, and if the MDE you would need at four weeks is larger than any realistic business impact, you should not run the test, because the site simply does not have the traffic. Choosing your MDE target is a business decision: what improvement actually generates meaningful impact, and is the revenue gain worth the cost of building and running the test at all.
Once running, a test is done only when three conditions are all met, the holy trinity of significance at ninety-five percent or better, statistical power of at least eighty percent, and the pre-calculated sample size reached. There are no magic numbers that override this. Hitting a hundred conversions per variant means nothing, the tool declaring a winner means nothing on its own, and ninety-five percent significance alone is necessary but not sufficient. On top of the trinity, the test must span multiple business cycles, typically two to four weeks, because Monday visitors behave differently from Friday visitors, mornings differ from evenings, and holidays, competitor sales, and seasonality all distort a single short window. A convenient sample is not a representative one. Start every test at four weeks, and only after a track record of consistent three-week results should you consider shortening, never below two.
Two errors haunt testing. A Type I error, the false positive, is believing B beat A when it did not, almost always caused by stopping too early on too little data. A Type II error, the false negative, is missing a real winner because of insufficient power. The false positive is the more common and more dangerous, the imaginary winner that flattens out with more data, and acting on one can send an entire company in the wrong direction on the strength of noise. It helps to know what significance actually means, because most people get it wrong. The p-value is not the probability that B is better than A. It is the probability of seeing a result this extreme assuming A and B are actually identical. You do not need to internalise the philosophy, only the rule: require a p-value of 0.05 or less and that is sufficient for practical use, while raising the bar to ninety-nine percent slashes false positives at the cost of far more traffic.
There is also a more intuitive school. Where frequentist statistics deals in p-values and significance, Bayesian statistics reports the probability that B beats A, which is far easier to explain to stakeholders. Saying there is a seventy percent chance B is better lands in a way that p-values never will, and it supports a useful kind of decision: if B has even a fifty percent chance of beating A by fifteen percent, the expected value favours switching, because the downside is staying where you are and the upside is real. It is the same logic as a bet you win seventy percent of the time, which you would take all day.
A couple of related tools round this out. Bandit testing is not a replacement for A/B testing but a different instrument for a different goal, shifting traffic toward the winner in real time to maximise revenue during a short, time-limited promotion where you cannot wait four weeks. Multivariate testing varies many elements at once to understand how they interact, but it needs enormous traffic, well into six figures of monthly users, so you should default to A/B testing and reach for multivariate only when interaction effects are specifically what you need to learn.
What you measure matters as much as how. Your primary metric should be a business outcome, revenue, transactions, trials, sign-ups, never a micro-conversion like add-to-cart or click-through. Getting more people to add to cart does not mean more people buy, and optimising the micro-metric can simply push the problem downstream while you congratulate yourself. The common objection, that you do not have enough purchases to use as a metric, is not an exception but a stop sign: if you cannot measure a business outcome, do not run the test.
A few operational hazards remain. Running two tests on the same page at once is best avoided, because even if you exclude overlapping users you still cannot see how the changes interact, and shipping both winners can create a conflict, so run them sequentially. Cookie-based testing also degrades over time as users clear cookies, switch devices, and get attributed across browsers, which is exactly why four weeks is the sweet spot and why a six-month test, with pollution potentially reaching thirty percent, can be meaningless. And never trust the testing tool alone, since tools optimise for engagement with their own product and ignore business cycles, pollution, and power. Always run your own post-test calculation, cross-reference with your analytics, and segment the data to understand how behaviour changed, not just that it did.
Testing strategy
With the mechanics in place, strategy is about where to aim. Picture a pyramid you climb from the bottom. The base is low-hanging fruit, known problems with fairly obvious categories of solution, which you still test because you do not know which specific version wins. Above that sits creative and persuasion testing, scarcity, social proof, competitive UX ideas, explored once the obvious problems are exhausted. At the top sits innovative testing, full redesigns and repositioning, the highest risk because it can fail catastrophically and the highest ceiling because it is the only route to doubling or tripling conversion rates. You do not start there.
On the site itself, start closest to the money. The homepage has high traffic but a low conversion rate because visitors are still evaluating, so it needs a large sample. The checkout has less traffic but a very high conversion rate because the people who reach it have already done the hard work of choosing, so a smaller sample can detect a lift. That makes checkout a more efficient place to find wins than its traffic alone suggests.
One piece of common advice deserves direct contradiction. You will hear that you should always test one change at a time so you know what caused the result. That is correct for Google and wrong for almost everyone else, because small changes produce small effects, small effects need huge traffic to detect, and most sites cannot see a one percent lift, they need ten to thirty percent. The right principle is to test one hypothesis at a time, not one change at a time. If your hypothesis is about trust, change all the trust-related elements together, the testimonials, the logos, the guarantees, the badges, in a single variation. If it wins, trust drives conversion on that page, and if it loses, trust is not the lever, and either way you learn something actionable. This is the difference between iterative testing, the research-driven default that delivers consistent incremental gains at low risk, and innovative testing, the big swing you reserve for when you have hit a local maximum and nothing incremental moves the needle anymore.
When a test loses, the response depends on the research behind it. If the data was strong and clearly pointed at a real problem, then the solution was wrong, not the problem, so you iterate and dig deeper, the way a mobile page that lost despite good copy went on to win by eighty percent after being shortened and having its form fields reordered. If the data was weak and the hypothesis was largely a guess, accept the loss as a low-confidence bet and go back to research.
Finally, guard against the validity threats that can make a winning idea look like a loser or the reverse. The most common is the instrumentation effect, where the variant is simply broken, the CSS wrong on mobile, the JavaScript not firing, the button unclickable on a tablet, which is why you QA on every device and browser before launch and why mature teams employ dedicated QA. The second is selection bias, where buying traffic to speed a test up brings in an audience that responds to the creative differently from your real visitors, so test only on representative traffic. The third is the history effect, where an external event, a competitor’s sale, a news story, a holiday, drives the result rather than your variant, so stay aware of the context around your test window and extend the test if something unusual disrupts it.
The foundations in one breath
Strip it all back and the foundations are simple to state and hard to practise. A/B testing measures whether a change produces real business impact, so spend most of your effort on research and only a little on the test itself. Prioritise with a framework that rewards evidence over opinion. Respect the statistics, the holy trinity of significance, power, and sample size, run for at least four weeks across full business cycles, measure a business outcome rather than a micro-conversion, and never trust the tool alone. Test one hypothesis at a time with changes big enough to detect, climb the pyramid from low-hanging fruit toward bold redesigns only when you must, and QA everything before it goes live. Do those consistently and the compound interest takes care of itself.
See you soon.
[…] A/B Testing Basics […]