A free Excel checklist for QA-ing experiments, with pre-live and post-launch checkpoints.
Most conversion teams have a testing process. Fewer have a QA process. The result is predictable: tests go live with a broken variant on Safari, an event that fires twice, or a split that quietly sends 60% of traffic to control. The test still produces a number at the end. That is the dangerous part.
A broken test does not fail, it lies
When a landing page breaks, you find out. Traffic drops, someone complains, you fix it.
When an experiment breaks, you usually do not find out. It runs its two weeks, reaches significance, and hands you a result. If the variant rendered badly for a quarter of your mobile users, that result is real data about a broken page, not about the idea you meant to test. You then ship the losing variant, or kill a winning idea, and you carry that wrong conclusion into every roadmap decision that follows.
That is why experiment QA is worth more than page QA. A page bug costs you a day of conversions. A test bug costs you a decision.
Why QA does not stop at launch
The usual approach is a single sweep before the test goes live. That catches rendering and setup problems, and it is necessary, but it misses the whole category of failures that only appear under real traffic.
Tests break after launch for reasons that pre-live QA cannot see. A browser version you did not have to hand renders the variant wrong. An analytics event stops firing after a deploy on Thursday. The traffic split drifts because a bot filter changed. A third-party script starts racing your test and users see a flash of the original page.
So the checklist repeats at four checkpoints: pre-live, 24 hours, 72 hours, and 7 days. The pre-live sweep is the full one. The rest are lighter re-runs, and their entire purpose is to catch the quiet failures while the test can still be saved. A problem found at 24 hours costs you a day. The same problem found at the readout costs you the whole test.
Two reviewers, not one
Every checkpoint has two columns, A and B, for two independent reviewers.
This matters more than it sounds. The person who built the test knows how it is supposed to work, which means they unconsciously use it correctly. They click the path they designed. They do not try the thing no sensible person would try, which is exactly what real users do all day. A second reviewer with no investment in the implementation finds things the builder cannot see.
If you genuinely only have one reviewer, use column A and mark B as N/A. But treat that as a compromise rather than the default.
What the checklist covers
48 checks across six sections:
- Setup and targeting. Audience rules, traffic allocation, URL targeting, preview mode for every variant, exclusions for internal traffic and overlapping tests.
- Browser and device. Chrome, Safari, Firefox and Edge on desktop, Safari on iOS, Chrome on Android, responsive breakpoints, and behaviour on a throttled connection.
- Function check. Rendering, flicker, links, forms, interactive components, click zones, console errors, layout shift, back and forward navigation.
- Data and segmentation. Split allocation, variant naming, primary and guardrail metrics, duplicate events, and whether the data actually arrives somewhere you can query it.
- Content and accessibility. Proofreading, alt text, keyboard operability, focus states, contrast, screen reader announcements.
- Compliance and legal. Offer qualification, terms one click away, regulatory logos, age gating, consent banners, opt-out routes, and the accuracy of any claims or pricing.
Section 6 is where regulated industries add their own requirements: financial promotions rules, gambling licence conditions, pharmaceutical claims. The rest applies to almost any test.
How to use it
Save a fresh copy for each test and name it after the test. Fill in the header, then work down the checks, marking each one OK, Issue, or N/A.
The workbook counts open issues per checkpoint and shows a verdict: BLOCKED while any issue is open, IN PROGRESS while the first reviewer still has blanks, and PRE-LIVE PASSED once every pre-live check is marked and none is an issue. Anything marked Issue gets a row on the Issue Log sheet, with enough detail that someone else can reproduce it.
The completed file is your audit trail. It records who checked what, when, and what was wrong. Six months later, when someone asks whether that surprising result was trustworthy, you will have an answer.
Hope this helps, Andrei.
