Most teams that get into experimentation start in the same place: they run tests. They pick a page, build a variant, ship it, and read the result. That is the visible part of the work, and it is also the smallest part. The programs that survive and grow are the ones that treat testing as the third stage of a larger operating system, not the whole of it. This article is about that operating system, the strategy that justifies it, the structure that runs it, and the culture that keeps it alive long after the person who built it has moved on.
Two arguments you have to win
The hardest part of experimentation is rarely the statistics. It is convincing the people around you, and that takes two different arguments aimed at two different audiences.
The first argument is for your peers, the marketing and product leaders who will use the program. It runs roughly like this. We live in a world drowning in data, and companies either have a framework for turning that data into action or they drown in analysis paralysis. The new game is applying the scientific method to business decisions, and the experimentation program is the capability that makes it cheaper, more accessible, and faster than ever before. The proof is all around us: the randomised controlled trial, once confined to medicine, is now mainstream, and you are exposed to hundreds of live experiments every day without noticing.
The second argument is for the person who controls budget and headcount, usually the CEO, and it has to be about the organisation, not the tests. The leading companies experiment openly, it is public knowledge, and it attracts the best talent. Winners build cultures of innovation and autonomy while losers are left with the people nobody else wanted. The deeper shift is democratised decision making, pushing decisions down from the C-suite to the product owners closest to the customer, and experimentation is the mechanism that makes that safe. This argument matters because a program that wants to grow beyond marketing into product, pricing, and operations needs the backing of the person who funds it, not just the person who runs the tests.
Why more data is not the answer
There is a comfortable assumption that more data leads to better decisions. It is only half true, and the missing half is what makes experimentation necessary. The path from raw data to a decision runs through information, then knowledge, and at every step human intuition does the pattern-finding. The trouble is that human pattern-finding is systematically biased. We notice fast changes and miss slow ones, we see what confirms what we already believe, we build whole stories from scraps of evidence, we assume we know what other people are thinking, and we round probabilities into something mentally convenient. Rory Sutherland’s line captures it: people run on logic about as much as a horse runs on petrol. A restaurant called Studio 97 outsells one called Studio 17, wine tastes better from a heavier bottle, and a painkiller works better when we believe it was expensive. Observed data cannot escape the intuitive layer we apply when we interpret it, and that is a structural fact about how humans process information, not a failure of data collection.
Observational data has three further limits worth naming. It generalises poorly, because the more novel a situation is, the less the past can tell you about it. It transfers poorly, because what works in one market, geography, or moment does not reliably hold in another. And most fundamentally, it is associative, showing what co-occurs rather than what causes what.
That last point is where Judea Pearl’s ladder of causality makes the case for experimentation precise. The bottom rung is seeing, pure observation, which answers whether X correlates with Y. The middle rung is doing, intervention, which answers what happens to Y if you actually change X. The top rung is imagining, the counterfactual, which asks what would have happened otherwise. A/B testing lives squarely on the middle rung. It is an intervention, you change the environment and measure the causal effect, and that is categorically different from observation. No amount of analysis on the bottom rung, however sophisticated, including machine learning on big data, can climb to it. Analytics tells you what is happening. Experimentation tells you what causes what.
Getting the words straight
It helps to be precise about terms that often get used interchangeably. Experimentation is the whole operating system: program and project management, finding problems, testing solutions, and making decisions. A/B testing is a specific tactic within it, the randomised controlled trial applied online. Conversion rate optimisation is a mindset and a channel focus, optimising toward conversion as the primary driver metric, while customer experience optimisation is its cousin focused on experience quality and metrics like NPS. A/B testing is a subset of experimentation, CRO is a channel-specific application of it, and experimentation is the operating system that contains them both.
A mature program pursues three goals at once: revenue, through outcome metrics; customer understanding, through behaviour and perception; and process, through the efficiency and velocity of the team itself. It delivers three kinds of benefit too. It reveals mechanism, the why behind an effect rather than just the fact of it. It quantifies magnitude and trade-offs, such as whether a short-term conversion lift quietly costs you lifetime value. And it mitigates risk, validating that a change does not break existing behaviour before full rollout, which is what most tests in practice are actually for.
The operating system: four stages, not one
The core model is a pyramid of four stages, each resting on the one below. Stage one is assessment and integration, identifying the right problems and connecting them to strategy. Stage two is process and planning, prioritising and roadmapping solutions. Stage three is test and learn, the running of experiments. Stage four is decisions and execution, making and communicating the decisions that result. Most frameworks obsess over stage three and ignore the rest, which is exactly why so many programs fail.
Two ideas run through every stage. An artifact is something you create and reference, a template, a document, a model. A ritual is something you do on a cadence, a workshop, a review, a meeting. Each stage needs both, the tools and the activities, and a program is only as healthy as its weakest of the two.
The classic failure mode is starting at stage three without stage one, which produces what you might call the whack-a-mole problem. Tests get run that do not connect to strategy, the website slowly turns into an arcade game of disconnected wins, and eventually leadership cannot see the link to business objectives and the program dies. The cure is the foundation that impatient teams skip.
Stage one: are we working on the right problems?
This stage exists to answer one question that programs almost never ask out loud: are we even working on the right problems? The temptation is always to leap to solutions, so this stage deliberately forbids them and stays on problems alone.
Four artifacts do the work. Strategy maps trace the line from business mission down through product strategy and marketing strategy to the experimentation program, so that what you test actually serves where the business is going. Misalignment here is genuinely dangerous: picture a cosmetics retailer where the CEO wants conversion rate as the headline metric while the owner wants to build luxury perception and optimise for lifetime value. A single program cannot serve both, and left unresolved the conflict poisons every roadmap and every decision. Goal tree maps then connect company-level North Star metrics down through growth-area metrics to the driver metrics a single experiment can move, plus the guardrail metrics that protect system integrity. The Airbnb example makes the stakes vivid: the same driver metric, conversion rate, sitting beneath revenue, nights booked, or unique users booking produces three entirely different testing roadmaps. Choose the goal metric before you plan a single test. Data maps document where data lives, how it flows, and what connections are missing, because only when experiment variant IDs reach the warehouse and join to CRM data can you answer the questions that matter, like which traffic sources produce the highest lifetime value. Research maps document the qualitative and quantitative sources you draw on, analytics for where and how much, user testing for behaviour under friction, polls and chat logs for fears and doubts, surveys for voice of customer at scale, and expert review for known principles.
The output of all that research is not a list of fifty issues. It is at most three problem themes, each with a clear problem statement, the strategic activities to address it, candidate solutions, the metrics it touches, and the data that justifies it. The rituals that sustain the stage are quarterly problem-focused workshops where solutions are explicitly banned, ongoing research on a defined cadence, and quarterly business reviews that keep asking whether the program is still aimed at the right problems.
Stage two: how do we prioritise and roadmap?
Once you know the problems, you have to choose, prioritise, and schedule the solutions, and decide who owns what. Prioritisation works on three levels. First, should you test this at all? If there is no downside, just build it. If there is no real hypothesis, no business-meaningful metric, or no action that would actually change based on the result, it does not belong on the roadmap. Second, where and how can you test, which is a page-by-page power analysis asking what effect size you could detect given the traffic, and what that effect is worth. Third, what to test, scored with a binary or ordinal model rather than subjective one-to-ten scales. Scoring impact, confidence, and effort as yes-or-no questions avoids the endless “is this a seven or an eight” debate, because two people almost always agree on yes or no, and it produces a defensible queue. It is also the cleanest way to handle the highest-paid person’s pet idea: score it with the same model, and if it lands at five out of sixteen while your top item scores fourteen, the conversation becomes data-driven rather than political.
Three more artifacts support the stage. A workflow system has three layers that must not be confused: a task layer for who does what by when, a data layer that is a searchable relational record of every test and its metadata, and a communication layer for getting the right information to the right person. The critical rule is never to force a task tool like Jira to act as the data layer, because a to-do list and a relational database are structurally different and forcing one to be the other produces chaos. A toolbox of statistical calculators handles sample size, business case, and sample ratio mismatch, the last of which checks that traffic was actually split as intended and should be a standard step at every launch. And an organisational model plus a RACI defines ownership, with most mature programs aiming for a centre of excellence that sets standards and supports teams who run their own tests, the middle path between a slow central bottleneck and an inconsistent free-for-all.
Stage three: run the test properly
This is the part everyone already knows, so the discipline is in the structure around it. A workflow map defines the standard operating process before, during, and after a test goes live, covering approvals and conflicts, the QA checklist and SRM monitoring, the in-flight checks and stop triggers, and who reviews the result and how. Build it with the team, start rough, and embed it where the work actually happens rather than in a wiki nobody opens.
The central artifact is the test document, the single most important operational object in the program. It holds the entire story of a test from origin to conclusion, usually as a slide deck because slides are shareable by link, extractable into larger reports, and easy to circulate. It runs from the originating insight and problem theme, through the test plan and business case, the design and dev specs, the QA log, and the full metrics, to the one slide that matters most: the scorecard. The scorecard carries the test name, the hypothesis, a control-versus-variant image, the outcome, and the key takeaway, and it has to be digestible in under a minute because it is what gets posted to Slack and shown in the town hall. Brand it for the client’s internal audience, not the agency’s, because that is who reads it.
Around the document sits the phase-gating system, a set of one-way checkpoints a test cannot pass until it qualifies. Is the problem a strategic priority with a measurable hypothesis? Has tech signed off and confirmed no conflicts? Are the specs complete and the RACI set? Has QA passed across browsers and devices with SRM monitoring in place? Has the test run long enough to reach its minimum detectable effect? Does the result meet the success criteria defined in advance? The document holds the evidence and the gates enforce the process, which together stop designers polishing while the lead just wants the hypothesis tested, and stop half-QA’d variants going live.
Stage four: decide, and make the decision stick
The most painful failure in experimentation is not a losing test, it is a winning one that gets thrown away. There is a well-known story of a landing page that tripled sign-ups after a year of work, only to be reverted by executives who disliked how it looked the moment its creator left. Knowing what works is useless if you cannot make the change and make it last. That is what this stage protects.
It starts before launch, with a premortem that defines what happens in every possible outcome. For a superiority test, seeking a win, you implement on a positive result and not otherwise. For a non-inferiority test, seeking only to avoid harm, a flat result is the goal and you implement on flat or positive. Crucially, the premortem must also specify the decision rule for the borderline cases in advance, the flat primary with a positive secondary, the positive primary with a negative secondary, because deciding those after the fact creates delay and conflict. The premortem pairs with an implementation checklist covering scope, stakeholder buy-in, the technical work and its owner, how impact will be confirmed, and what the team actually learned about customer behaviour and the mechanism behind it.
Confirming impact after rollout has several methods, each with trade-offs: a test-specific holdout for precise attribution but hard maintenance, a permanent continuous holdout that is easier to keep but blunter, a flip test that re-tests the control to prove repeatability, and a time-series check using moving averages for a quick if less precise read.
Then comes communication, which is deceptively hard. The principle is the right information, to the right person, at the right time, in the right format, and the mechanism, as with comedy or advertising, is context. Communication runs on a spectrum from enriched to simplified. The enriched end, full metadata and statistical tables in the data layer, feeds institutional knowledge. The simplified end, the thirty-second scorecard and the Slack post, spreads the culture. Both are necessary. A few habits make communication land: tell customer stories about what behaviour or perception changed rather than reciting p-values, be humble and share the threats to validity because hiding flaws erodes trust faster than admitting them, give credit publicly because engagement follows recognition, and automate distribution so scorecards post themselves when a test launches or concludes, then optimise those messages the way you would optimise a page.
Scaling: trust, maturity, and the metrics of the program itself
Culture is the thing that finally makes a program durable, and it rests on three kinds of trust. Trust in the system, meaning data integrity and guardrails the team believes in. Trust in the team, meaning the psychological safety to report errors and lose tests without fear. And trust in the goal, meaning a clear strategic direction with honest trade-offs. A revealing corollary: a healthy program has a visible error rate, roughly five to ten percent. If errors never surface, either the team is afraid to admit them or the ideas are too timid to be worth testing, and both are bad signs.
Programs can begin from any of three orientations, which also work as phases. Learn to test, building trustworthy systems. Test to learn, exploring customer problems. Test to win, exploiting insights for revenue. The healthiest programs move through all three in order, and jumping straight to winning without first earning reliability produces results nobody trusts. As a program matures, tests should grow in both volume and complexity, from simple client-side copy changes by one team toward server-side tests, model tests, and personalisation across many teams.
Several maturity models help, each for a different purpose: Thomke’s aware-believe-commit-diffuse-embed ladder as a conversation starter, the crawl-walk-run-fly model as a capability benchmark, and an audit of strategy, people, process, and tools that uniquely measures how different functions perceive the same program, since a central team and a product team often see it very differently. Whichever you use, the point is to locate where you are against where you could be and to plan the drivers that close the gap.
Finally, measure the program, not just the experiments. Guardrail metrics span velocity, how fast insights become launches and decisions; quality, the rate of useful learning and the balance of the portfolio; efficiency, where errors occur and how fast wins ship; effectiveness, the decision rate and the share of changes driven by evidence rather than gut; trust, the win, fail, and error rates; and culture, how many teams are actively testing. The win rate deserves special care: tracked alone it misleads, because a win rate far above the fail rate means you are only testing safe bets, and a zero error rate means people are hiding mistakes. A healthy program shows visible failures and visible errors. IBM’s journey makes the payoff concrete, going from a single gatekeeper running ninety-seven tests a year to a centre of excellence running three thousand tests a year with thousands of employees involved, simply by driving the cost of a test down until volume rose and, with standards held, quality rose with it.
The principles that hold it together
Strip everything back and a few principles remain. Keep lowering the cost of running an experiment so more people can use the system. Prefer sets of experiments to single bets, because testing your way into a win beats gambling on one idea. Keep perspective by connecting driver metrics up to lag metrics and strategy rather than fixating on the local number. Invest in the technical and statistical infrastructure, because study design and integrity are not optional. And run experiments ethically, testing with your customers rather than on them, treating experimentation as a way to understand and serve people rather than to exploit them. Get those right, build the four stages rather than just the third, and you end up with something rare: a testing program that does not depend on any one person to survive.
See you soon.
[…] Experimentation Program: Building a Testing Program That Lasts […]