A one-page-per-stage checklist covering everything to decide, build, verify, run and analyse so that an A/B test produces a result you can trust: 88 checks across twelve stages, from writing the hypothesis to removing the code after rollout.
Most invalid A/B tests are not invalidated during the test. They are invalidated before it starts, by a decision nobody wrote down. The sample size was never calculated, so the test ends when someone loses patience. Randomisation happens per session rather than per user, so the same person sees both variants. Exposure fires on page load rather than at the point of exposure, so half the sample never saw the thing being tested. Each of those produces a number, and the number looks exactly like a result.
This checklist is built around that: stages 01 to 08 are pre-launch and are where almost every broken test is created, and stages 09 to 12 are the discipline that keeps a live test valid and turns the outcome into a decision. Twenty-eight items are marked as highest leverage, chosen because they invalidate a test outright rather than merely weaken it, so there is a short version when there is no time for the long one.
It is tool-agnostic and works the same for client-side, server-side and feature-flag testing. Print it, or keep it open next to the test plan and tick as you go.
Thanks, Andrei.
