The A/B Test Dataset: Five Faults, and Why the Answer Is Not “Ship It”

A four week test on a checkout change has finished. The product manager has seen a dashboard showing the variant winning on mobile and wants to ship on Monday. You have 39,924 rows and until the end of the day.

The correct answer is no, and not because the variant lost. The result cannot be read at all. Here is how you establish that in about ten minutes, and the five faults that are in the file.

If you would rather work it out yourself, the download includes a brief with the task and nothing else. Stop here and come back afterwards.

What you are given

One row per assignment: user id, assignment timestamp, variant, device, browser, country, whether they converted, revenue, and page load time. Four weeks, two arms, nine columns.

1. Check the split before anything else

The test was configured 50/50. It arrived 20,412 control against 19,512 variant.

from scipy import stats
counts = df["variant"].value_counts()
expected = len(df) / 2
chi = sum((c - expected) ** 2 / expected for c in counts)
stats.chi2.sf(chi, 1)      # 6.66e-06

That is a sample ratio mismatch, and the threshold for alarm is 0.0005, not 0.05. You run this check on every test you ever look at, so a 5% threshold would raise a false alarm on one test in twenty and you would learn to ignore it. At one in 150,000 this is not chance.

Everything below this line is diagnosis. The result itself is not interpretable, and no amount of segmenting will make it so.

2. The cause is traceable, but only if you look at counts

As shares, Safari is 24.9% of the control and 11.4% of the variant. It is tempting to stop there and call it a redirect failing on Safari.

Look at the counts instead and that story collapses.

Chrome went from 5,156 to 10,076, which is 1.95 times. Edge fell to 0.65 times, Firefox to 0.76, Safari to 0.44.

Chrome nearly doubled. A redirect that fails for Safari users cannot add Chrome users, so this is not a delivery failure at all. The assignment itself is correlated with browser, most likely a hash of something that is not independent of the user agent.

This is the most useful habit in the article. Shares tell you a mix changed. Only counts tell you which direction it moved, and therefore what kind of bug you are looking for. Normalising too early hides the one number that identifies the cause.

pd.crosstab(df["variant"], df["browser"])              # counts, look here first
pd.crosstab(df["variant"], df["browser"], normalize="index")   # shares, second

3. Three hundred users in both arms

312 users appear as both control and variant, out of 39,612 unique users.

df.groupby("user_id")["variant"].nunique().gt(1).sum()   # 312

A cookie reset, a cross device journey, or an assignment bug. Their outcomes are counted on both sides, which dilutes any real difference toward zero. On its own this would not sink a test. Alongside a sample ratio mismatch it points at the same place: assignment is not doing what you think.

4. The segment that looks like a win

Tablet Chrome: control converts at 1.54%, variant at 4.00%, p = 0.026. A 160% lift. This is the slice on the dashboard.

It is not real. There is no true effect anywhere in this dataset, in any slice. Overall the two arms convert at 2.83% and 2.75%, p = 0.63.

What you are looking at is multiple comparisons. Three devices by four browsers is twelve tests, and at a 5% threshold you expect roughly one of twelve to come up significant when nothing whatsoever is happening. Exactly one did.

Three tells, in order of usefulness:

The slice is small. 390 control users and 650 variant. A 160% lift on a thousand users is not a finding, it is a coin landing heads six times.

It does not survive a correction. Bonferroni across twelve tests puts the threshold at 0.0042. The slice’s 0.026 is nowhere near it.

You went looking for it. It was not in the hypothesis. That is the entire difference between analysis and hypothesis generation.

The honest sentence is “this is worth a dedicated test”. It is never “the test won on tablet Chrome”.

5. Three orders move the revenue metric

Three orders over £3,000 sit in the variant: 3,650, 5,220 and 5,220. The next largest order anywhere in the file is £183.

Mean revenue per user in the variant arm is 2.75 with them and 2.03 without, a 36% swing from three rows out of nearly forty thousand. Those three orders are 26% of everything that arm earned.

The habit: on any revenue metric, look at the top ten values before you look at the mean. Cap at a stated percentile, and say in the write up that you capped and where.

6. The guardrail got worse

Mean page load is 1,174ms in the variant against 923ms in the control, a 27% regression. Even had the test been readable, that is a stop condition on its own, and it is a plausible cause of a real negative effect that this broken assignment would have hidden.

The two sentences to send

The split came out 20,412 against 19,512 when it was configured 50/50, which has odds of about one in 150,000, so the groups are not comparable and I cannot read the result either way. Assignment looks correlated with browser rather than random, since Chrome doubled in the variant while every other browser fell, and I would fix that and re-run before we read anything into the segments.

The order that matters

Split, then integrity, then the primary metric, then guardrails, then the segments you registered in advance. Every fault in this file is findable in the first two steps, and the dashboard that started the conversation came from the last one. Doing the segments first is how a broken test ships.

Have fun, Andrei.