Post AB test analysis

In the Real World · Brief · Data science · Core · 1 to 2 days

A test finished last night. Someone has to say ship or do not ship, today. You have the raw assignment log and nothing else. This is the brief, not the answer: work it yourself before reading the walkthrough.

The situation

Thornbury Home is a homewares retailer selling into the UK and Ireland. It redesigned its product detail page: larger imagery, the delivery promise above the fold, and the add-to-basket button pinned on mobile. The redesign ran against the current page for 28 days, 2026-05-04 to 2026-05-31, with traffic configured to split 50/50.

The growth lead has already seen the experiment dashboard and wants to roll the redesign out across 9,000 product pages. Finance wants the revenue number stood up before the engineering time is committed. You have been handed the raw log.

Your job

Produce a decision memo that a non-analyst can act on. It has to say ship or do not ship, give the evidence, and state anything that has to be fixed before the question can be asked again.

The data

One file, thornbury-ab-test.csv, 39,924 rows. It is the raw assignment log with the outcome columns joined on. It has not been cleaned or filtered for you.

What the team agreed before the test started

The hypotheses on record were that the redesign lifts conversion, that the lift is larger on mobile, that revenue per user rises with it, and that page load is unchanged.

Constraints

• Assignment is client side. The experiment script runs in the visitor’s browser.
• The export joins assignment records to orders in the warehouse.
• Baseline conversion is around three percent.
• Trading confirm no promotions ran in the window.
• The decision is needed today. You cannot commission new data.

Definition of done

You are finished when you can hand over a memo containing all six of these.

1. A ship or do not ship recommendation, stated in the first sentence.
2. The primary metric result with a confidence interval, not just a p-value.
3. An explicit statement on whether the test was valid, with the check that supports it.
4. The secondary metric, reported in a way that one unusual order could not have produced.
5. The guardrail verdict.
6. A list of anything engineering or data engineering has to fix, with the evidence for each.

If you want to go further

• Work out what effect size this test was capable of detecting at all, and compare it to what a page redesign plausibly delivers.
• Write the monitoring rule that would have caught the problem while the test was still running, rather than after it finished.
• Specify the rerun: how long it would need to run, and what would have to be true before starting it.

In the Real World · Data science · Core · 1 to 2 days

The test finished on the Sunday. By Monday morning the growth lead had a number, and it was a good one: revenue per user up 32.3%. The rollout slide was already written. This is the analysis that stopped it, and the four faults it found in a test that looked clean from the dashboard.

Section 1Problem Definition

No code yet, and that is deliberate. Almost every bad test decision is made before anyone opens the data, by failing to write down what would count as a win. Once you have seen the numbers it is far too late to decide what you were measuring.

Business objective

Thornbury Home redesigned its product detail page: larger imagery, the delivery promise moved above the fold, and the add-to-basket button pinned on mobile. The design team expects more people to reach the basket. Finance wants to know whether it is worth the engineering time to roll out across 9,000 products.

Problem statement

Decide whether the redesigned page should replace the current one, using 28 days of randomised traffic between 2026-05-04 and 2026-05-31.

Success metrics, written before the data is opened

Hypotheses

Four, written down now and tested in section 4, rather than invented afterwards to explain whatever the data turned out to show.

1. H1. The redesign lifts conversion rate.
2. H2. The lift is larger on mobile, because that is where the pinned button changes the most.
3. H3. Revenue per user rises at least in line with conversion.
4. H4. Page load is unchanged. The guardrail holds.

The decision rule

Also written before looking. Ship if the primary metric is up with a p-value under 0.05 and the guardrail holds. Do not ship if the primary is flat or down. And, the clause people forget: if the assignment mechanism fails its validity checks, there is no decision to make at all, because the two groups were never comparable and no amount of analysis fixes that.

Risks and constraints

• Assignment is client-side, so any browser that blocks or delays the experiment script can end up in the wrong arm, or in no arm.
• The export joins assignment records to orders. Joins fan out when a key is not unique.
• Baseline conversion is low, near three percent, which means the test needs far more traffic than people expect. Section 7d puts a number on that.
• Trading ran no promotions in the window, so seasonality is not a confounder here.

Stakeholders

The growth lead wants the rollout. Engineering owns the assignment script and will have to fix whatever is broken. Finance wants the revenue number, and is the reason the winsorised metric exists rather than the raw one.

Section 2Data Collection

One export from the experimentation warehouse: one row per assignment, with the outcome columns joined on. Read the dictionary before the data, because it tells you what the grain is supposed to be, and the first real finding in this project is that the data does not respect its own grain.

# the analysis stack, nothing exotic
import numpy as np                       # numerical helpers
import pandas as pd                      # dataframes
from scipy import stats                  # the statistical tests

pd.set_option('display.max_columns', 20) # show every column when we print

# parse_dates turns the timestamp column into real datetimes at load time,
# so we never have to remember to convert it later
df = pd.read_csv('data/thornbury-ab-test.csv', parse_dates=['assigned_at'])

print(df.shape)                          # (rows, columns)
print(df['user_id'].nunique(), 'unique users')
df.head()

(39,924, 9)
39,612 unique users

Note those two numbers before going any further. 39,924 rows and 39,612 users. The export is supposed to be one row per user. It is not, and 312 rows are unaccounted for. That gap is fault one, and we come back to it in 3a.

The data dictionary

Section 3Data Preprocessing

On an experiment, preprocessing is not tidying. It is the validity audit. Every fault found here is a reason the headline number cannot be trusted, so this section decides the outcome of the project long before any test is run.

3aDuplicates and schema checks

Start where the row count disagreed with the user count.

# which user_ids appear more than once, and how many rows do they account for
dup_ids  = df.loc[df['user_id'].duplicated(keep=False), 'user_id'].unique()
dup_rows = df[df['user_id'].isin(dup_ids)]

print('duplicated users :', len(dup_ids))
print('rows involved    :', len(dup_rows))
print('rows per user    :', dup_rows.groupby('user_id').size().value_counts().to_dict())

# the question that matters: is the same person sitting in both arms
arms_per_user = dup_rows.groupby('user_id')['variant'].nunique()
print('users in BOTH arms:', (arms_per_user > 1).sum())

duplicated users : 312
rows involved    : 624
rows per user    : {2: 312}
users in BOTH arms: 312

Every one of the 312 duplicated users appears exactly twice, once in each arm. That has two very different possible causes, and they call for different fixes, so it is worth one more line of code to tell them apart.

# if the two rows differ only by the arm label, this is an export problem.
# if they differ in timestamp or behaviour, the same person was genuinely
# bucketed twice, which is a much worse bug in the assignment service.
cmp_cols = ['assigned_at', 'device', 'browser', 'country',
            'converted', 'revenue', 'page_load_ms']

identical = dup_rows.groupby('user_id')[cmp_cols].nunique().eq(1).all(axis=1)
print('identical apart from the arm label:', identical.sum(), 'of', len(dup_ids))

identical apart from the arm label: 299 of 312

299 of 312 pairs are the same visit twice with a different label attached. Same timestamp, same device, same load time, same outcome. A visitor cannot load one page in two states at one instant, so this is not double bucketing. It is a join fanning out against a non-unique key in the export, and the fix belongs to the data pipeline rather than to the experiment.

3bHandling categorical mess

The usual audit. Run it even when you expect nothing, because it costs one line and the alternative is discovering a stray label halfway through the analysis.

# for each categorical, compare the raw level count against the count after
# stripping whitespace and lowercasing. If those disagree, the same value is
# hiding under several spellings.
for col in ['variant', 'device', 'browser', 'country']:
    raw_levels   = df[col].nunique()
    clean_levels = df[col].str.strip().str.lower().nunique()
    print(f'{col:9s} {raw_levels} levels, {raw_levels - clean_levels} spelling variants',
          sorted(df[col].unique()))

variant   2 levels, 0 spelling variants ['control', 'variant']
device    3 levels, 0 spelling variants ['desktop', 'mobile', 'tablet']
browser   4 levels, 0 spelling variants ['Chrome', 'Edge', 'Firefox', 'Safari']
country   2 levels, 0 spelling variants ['Ireland', 'UK']

Clean. No stray casing, no trailing spaces, no duplicate labels. Reporting that plainly is part of the job: a reader needs to know the check ran, not just that nothing was said about it. Note the browser levels though, because that column is where this test comes apart in 4c.

3cDealing with outliers

The rule is that data errors get removed and genuine extremes get respected. Telling them apart is the skill.

# how far does the tail run, and who is in it
print('99th percentile order :', df['revenue'].quantile(0.99).round(2))
print('largest order         :', df['revenue'].max())
print('orders above 1,000    :', (df['revenue'] > 1000).sum())

# the arm those large orders landed in is the thing to look at
big = df[df['revenue'] > 1000]
print(big[['user_id', 'variant', 'browser', 'revenue']].to_string(index=False))

99th percentile order : 84.28
largest order         : 5220.0
orders above 1,000    : 3
 user_id variant  browser  revenue
 U0526961 variant     Edge  5220.00
 U0537470 variant   Chrome  5220.00
 U0530149 variant  Firefox  3650.00

Order values on a log scale. The typical order is around 73.31 pounds and the 99th percentile is 84.28. The three orders above one thousand are not in the same world as the rest of the distribution, and all three are in the variant arm.

Three orders above a thousand pounds, all 3 of them in the variant arm, and 2 of them at exactly 5220.00 pounds. Two different users, on two different browsers, three days apart, spending an identical amount to the penny. That is not customer behaviour. Real order values do not repeat exactly at the top of the tail.

The honest position is that we do not know what those rows are. A staff test order, a trade account, a currency field applied twice. What we do know is that they are the entire revenue result, which section 7c demonstrates, and that no rollout should ever rest on three rows whose provenance nobody can explain.

3dHandling missing values

# the missingness audit, in one line
print('missing cells :', df.isna().sum().sum())
print('affected cols :', [c for c in df.columns if df[c].isna().any()])

missing cells : 0
affected cols : []

Nothing missing anywhere. On a warehouse export of a running experiment that is normal and slightly suspicious at the same time, because it means the export has already applied some filter upstream. Worth a question to the pipeline owner. It is not a fault, so we move on.

3eHandling skewed data

# skew on all rows is dominated by the 97% of visits that spent nothing,
# so also measure it among the people who actually ordered
converters = df[df['revenue'] > 0]

print('skew, all rows   :', round(df['revenue'].skew(), 1))
print('skew, converters :', round(converters['revenue'].skew(), 1))
print('converters       :', len(converters))
print('mean order       :', round(converters['revenue'].mean(), 2))
print('median order     :', round(converters['revenue'].median(), 2))

skew, all rows   : 103.5
skew, converters : 19.5
converters       : 1,115
mean order       : 86.14
median order     : 73.31

A skew of 19.5 among converters, with a mean order of 86.14 against a median of 73.31. The mean sits well above the median, which is the signature of a few very large values pulling the average around. Any metric built on the raw mean of this column will be unstable, and section 5 is where we do something about it rather than just noting it.

3fData types and normalisation

# converted is stored as 0/1 integers, which is fine for arithmetic:
# the mean of a 0/1 column is the conversion rate, which is what we want
df['converted'] = df['converted'].astype(int)

# a calendar day column, for the day by day validity checks in section 7
df['day'] = df['assigned_at'].dt.date

# no scaling or normalisation here. Nothing in this analysis is distance based
# or gradient based, so rescaling would only make the numbers harder to read.
print(df.dtypes.to_string())

Worth saying explicitly, because the modelling reflex is to normalise everything: there is no model here. Scaling a column changes nothing about a proportion test and makes the output unreadable to the stakeholder who has to act on it.

Section 4Exploratory Data Analysis

On a modelling project this section explores the target. On an experiment it explores whether the randomisation actually happened, which is a different question and a more urgent one.

4aThe primary metric

# one row per arm: size, conversions, conversion rate, revenue and load time
summary = df.groupby('variant').agg(
    users      = ('user_id',      'size'),
    orders     = ('converted',    'sum'),
    conv_rate  = ('converted',    'mean'),
    rev_total  = ('revenue',      'sum'),
    rev_user   = ('revenue',      'mean'),
    load_ms    = ('page_load_ms', 'mean'),
)
print(summary.round(4).to_string())

         users  orders  conv_rate  rev_total  rev_user   load_ms
variant
control  20,412     578     0.0283  42,402.61    2.0773  922.7
variant  19,512     537     0.0275  53,641.91    2.7492  1174.2

Read that table the way the growth lead read it on Monday morning. Revenue per user goes from 2.08 to 2.75 pounds, a rise of 32.3%. That is the number on the slide.

Now read the two columns either side of it. Conversion rate went down, from 2.83% to 2.75%. Page load went up by 252 milliseconds. And the two arms are not the same size: 20,412 against 19,512, on a test that was configured 50/50.

Fewer people converting, but more revenue per head, on a slower page, in unequal groups. Those four facts do not sit together comfortably, and the rest of this analysis is the work of finding out why.

4bNumerical variables

# describe() on the three numeric columns, transposed so it reads down the page
print(df[['revenue', 'page_load_ms', 'converted']].describe().T.round(2).to_string())

# the guardrail, by arm
print()
print(df.groupby('variant')['page_load_ms'].describe().round(1).to_string())

         count     mean      std   min    25%    50%     75%      max
revenue  39,924  2.41      ...    0.00   0.00   0.00    0.00  5220.00
page_load_ms 39,924  1045.6   ...  120                        2187

           count    mean    std    min     25%     50%      75%     max
variant
control  20,412   922.7  259.5  120.0  748.0   923.0  1099.0  1905.0
variant  19,512   1174.2  262.1  120.0  998.0  1173.0  1350.0  2187.0

Page load by arm. The spread is almost identical, around 260ms in both, so this is not a few slow outliers. The entire distribution has moved right by roughly 252 milliseconds, which means every single visitor on the redesign waited longer.

This is the cleanest finding in the dataset and the easiest to miss, because guardrails live at the bottom of the dashboard. The quartiles move in lockstep: 748 to 998 at the 25th, 923 to 1173 at the median, 1099 to 1350 at the 75th. A uniform shift like that is a heavier page, not noise and not a handful of bad connections.

4cCategorical variables

Here is where the test falls over. The check is simple: for each level of each categorical, what share of that group landed in the variant arm. If assignment is random, every one of those shares should sit near 50 percent.

# share of each segment that landed in the variant arm, with a chi-square
# test of that split against the intended 50/50
for col in ['device', 'browser', 'country']:
    print('---', col)
    for level, sub in df.groupby(col):
        counts = sub['variant'].value_counts()
        c, v   = counts.get('control', 0), counts.get('variant', 0)
        p      = stats.chisquare([c, v]).pvalue     # expected is 50/50 by default
        print(f'  {level:8s} n={len(sub):6,d}  variant share {v/len(sub):.3f}  p={p:.4f}')

--- device
  desktop  n=12,018  variant share 0.489  p=0.0215
  mobile   n=25,115  variant share 0.491  p=0.0046
  tablet   n= 2,791  variant share 0.464  p=0.0002
--- browser
  Chrome   n=15,232  variant share 0.661  p=0.0000
  Edge     n= 8,329  variant share 0.395  p=0.0000
  Firefox  n= 9,050  variant share 0.433  p=0.0000
  Safari   n= 7,313  variant share 0.305  p=0.0000
--- country
  Ireland  n= 7,062  variant share 0.490  p=0.1108
  UK       n=32,862  variant share 0.488  p=0.0000

Device is mildly off. Country is fine. Browser is a wreck. Chrome sent 66.1% of its traffic to the variant while Safari sent 30.5%. Those are not close calls, and they are not the sort of thing that happens by chance in 15,232 and 7,313 visitors.

Browser mix within each arm. The control bars are the tell: 25.3% Chrome, 24.7% Edge, 25.1% Firefox, 24.9% Safari. Four browsers, a quarter each, spread of only 0.55 percentage points.

So the control group is not a random sample of Thornbury Home visitors. Something between the assignment service and this export is filling that arm on a rule that has nothing to do with real traffic. Until engineering says what, the two groups cannot be compared, because they are not made of the same people.

4dRelationships between variables

One relationship matters for the decision: whether the slower page is plausibly costing conversions, which would explain the direction of the primary metric.

# bucket load time and look at conversion within each bucket, inside the
# control arm only, so the comparison is not contaminated by the arm itself
control = df[df['variant'] == 'control'].copy()
control['load_bucket'] = pd.qcut(control['page_load_ms'], 5, precision=0)

print(control.groupby('load_bucket', observed=True)['converted']
             .agg(['size', 'mean']).round(4).to_string())

Within the control arm alone, conversion drifts down as load time rises. That is a correlation, not a causal estimate, and slow visits also skew toward older devices and poorer connections. But it means the 252 millisecond regression is not a cosmetic complaint. It points the same way as the primary metric, which fell.

4eTesting our hypotheses

The four from section 1, answered against the data rather than against the slide.

Four hypotheses, none of them supported, and the guardrail actively broken. Writing them down in advance is what makes that sentence possible. Without it, H3 becomes the headline and the other three are quietly never mentioned.

4fSubgroups

There will be pressure to find a segment where the redesign wins. Tablet obliges: 2.48% in control against 3.32% in the variant, which reads as a large relative gain.

It is not a finding. Tablet is 2,791 visitors producing a double digit number of orders, its own assignment split fails at p = 0.0002, and it is one of several segments examined after the fact. A subgroup that was not pre-registered, in a test whose randomisation is broken, is a coincidence with a chart attached.

Section 5Metric Construction

Section 3e showed that raw revenue per user is dominated by its tail. This is where we turn the three metrics from section 1 into definitions precise enough that two analysts would compute the same number.

# 1. PRIMARY: conversion rate. The mean of a 0/1 column, no construction needed.
#    Reported per arm.

# 2. SECONDARY: revenue per user, winsorised at the 99th percentile.
#    Winsorising CAPS extreme values at a threshold rather than deleting the
#    rows. The customer still counts as a converter, their spend is just not
#    allowed to swing the mean on its own.
cap = df['revenue'].quantile(0.99)
df['revenue_capped'] = df['revenue'].clip(upper=cap)
print('cap at the 99th percentile :', round(cap, 2))

# 3. GUARDRAIL: mean page load per arm, in milliseconds. Reported as a
#    difference, because the absolute number depends on the measurement point.

cap at the 99th percentile : 84.28

Three points about that cap, because winsorising is the step people argue about.

1. The threshold is chosen before seeing which arm benefits. The 99th percentile is a convention, set in section 1. Picking it afterwards, once you know it flatters your preferred result, is the same sin as picking the metric afterwards.
2. Capping is not deleting. The customer keeps their conversion. Only the amount is limited. Dropping the row would bias the primary metric to fix the secondary one.
3. It is a decision metric, not an accounting metric. Finance still books the full 5220.00 pounds. The capped figure exists to answer a different question: what does this page do to a typical visitor.

The uncapped figure is still reported, always, alongside the capped one. Hiding it would be its own kind of dishonesty. The two together are the finding.

Section 6Method Selection

Four choices, each with a reason, because the default in most post-test analysis is whatever the calculator on the intranet happens to do.

Why not a rank test on revenue

Mann-Whitney is the reflex for skewed data, and it is the wrong reflex here. It tests whether one distribution tends to sit above the other, which on a column that is zero for 97.21% of rows is almost entirely a restatement of the conversion rate. The business question is about pounds per visitor, so the mean is the quantity of interest. The fix for the tail is the cap in section 5, not a change of test.

The order the tests run in

Validity first, then the primary metric, then the secondary, then the guardrail. That ordering is not cosmetic. If the validity check fails, the remaining tests are computed and reported as diagnostics, never as results. Running them in the other order is how a broken test gets shipped: by the time anyone checks the split, the lift is already in a slide.

Section 7Analysis and Validation

7aThe baseline the business believes

Before running anything, write down what the room already thinks. On this test it is one sentence: the redesign lifted revenue per user by 32.3% and should be rolled out to all 9,000 product pages.

That belief has a specific source. Someone opened the experiment dashboard, read the revenue row, and stopped. It is not stupid, it is the number the tool puts in the largest font. The analysis has to beat that baseline, which means it has to explain why the 32.3% is not what it appears to be, not merely assert it.

7bThe analysis

Validity first

# the sample ratio mismatch check. Expected counts come from the CONFIGURED
# split, which was 50/50, not from the observed data.
counts = df['variant'].value_counts()
n_control, n_variant = counts['control'], counts['variant']
total = n_control + n_variant

chi2, p_srm = stats.chisquare([n_control, n_variant],
                              f_exp=[total * 0.5, total * 0.5])

print(f'control {n_control:,}   variant {n_variant:,}')
print(f'variant share {n_variant / total:.5f}   expected 0.50000')
print(f'chi2 {chi2:.3f}   p {p_srm:.8f}')

control 20,412   variant 19,512
variant share 0.48873   expected 0.50000
chi2 20.288   p 0.00000666

A gap of 900 visitors on a 39,924 visitor test, which is a variant share of 48.87% where 50 was configured. It looks like almost nothing. The p-value is 0.0000067.

That is the point of running the test rather than trusting your eye. A one percentage point drift is invisible on a dashboard and statistically impossible in a working randomiser. Something is deciding who goes where, and section 4c already showed what it correlates with.

Variant share by day against the intended 50 percent line. Only 2 of 28 days fail on their own at the 0.05 level, and none fail at 0.001. Pooled across the test, the same data gives p = 0.0000067.

The primary metric

# two-proportion z-test on conversion. Pooled variance under the null, which
# is the standard construction for a difference in proportions.
c = df[df['variant'] == 'control']
v = df[df['variant'] == 'variant']

p1, p2 = c['converted'].mean(), v['converted'].mean()
pool   = (c['converted'].sum() + v['converted'].sum()) / (len(c) + len(v))
se     = np.sqrt(pool * (1 - pool) * (1 / len(c) + 1 / len(v)))
z      = (p2 - p1) / se
p_val  = 2 * stats.norm.sf(abs(z))

# an unpooled Wald interval for the difference, which is what to report
se_ci  = np.sqrt(p1 * (1 - p1) / len(c) + p2 * (1 - p2) / len(v))
lo, hi = (p2 - p1) - 1.96 * se_ci, (p2 - p1) + 1.96 * se_ci

print(f'control {p1:.5f}   variant {p2:.5f}')
print(f'absolute {100*(p2-p1):+.4f} pp   relative {100*(p2/p1-1):+.2f}%')
print(f'z {z:.3f}   p {p_val:.4f}')
print(f'95% CI on the absolute difference: [{100*lo:+.4f}, {100*hi:+.4f}] pp')

control 0.02832   variant 0.02752
absolute -0.0795 pp   relative -2.81%
z -0.482   p 0.6298
95% CI on the absolute difference: [-0.4027, +0.2437] pp

No effect. The interval runs from -0.40 to +0.24 percentage points, comfortably spanning zero, and the point estimate is slightly negative. Read that interval rather than the p-value: it says the data is consistent with anything from a small loss to a small gain, which is a statement about how little this test learned.

The secondary metric, raw and capped

# the number on the slide, computed honestly with a Welch test
t_raw = stats.ttest_ind(v['revenue'], c['revenue'], equal_var=False)
print(f'RAW    control {c["revenue"].mean():.4f}   variant {v["revenue"].mean():.4f}'
      f'   {100*(v["revenue"].mean()/c["revenue"].mean()-1):+.2f}%   p {t_raw.pvalue:.4f}')

# the same comparison on the metric defined in section 5
t_cap = stats.ttest_ind(v['revenue_capped'], c['revenue_capped'], equal_var=False)
print(f'CAPPED control {c["revenue_capped"].mean():.4f}   variant {v["revenue_capped"].mean():.4f}'
      f'   {100*(v["revenue_capped"].mean()/c["revenue_capped"].mean()-1):+.2f}%   p {t_cap.pvalue:.4f}')

RAW    control 2.0773   variant 2.7492   +32.34%   p 0.1281
CAPPED control 1.8866   variant 1.8427   -2.32%   p 0.7030

The guardrail

t_load = stats.ttest_ind(v['page_load_ms'], c['page_load_ms'], equal_var=False)
diff   = v['page_load_ms'].mean() - c['page_load_ms'].mean()
print(f'control {c["page_load_ms"].mean():.1f}ms   variant {v["page_load_ms"].mean():.1f}ms')
print(f'difference {diff:+.1f}ms  ({100*diff/c["page_load_ms"].mean():+.1f}%)   p {t_load.pvalue:.2e}')

control 922.7ms   variant 1174.2ms
difference +251.5ms  (+27.3%)   p 0.00e+00

The guardrail fails, and it is the only unambiguous effect in the whole test. 252 milliseconds slower, 27.3%, with a p-value below the smallest number floating point can hold. Note the asymmetry: the assignment fault makes the conversion comparison untrustworthy, but it does not rescue this. A page that is heavier is heavier for everyone who loads it, whichever arm they were supposed to be in.

7cSensitivity and robustness

One result is worth nothing until you have tried to break it. Three attempts, each removing a different fault, to see whether any conclusion survives.

# 1. drop the duplicated users entirely and re-run the primary test
clean = df[~df['user_id'].isin(dup_ids)]

# 2. drop the three unexplained orders as well, and re-run revenue
clean_rev = clean[clean['revenue'] <= 1000]

Conversion is stable across every version, and it is stably nothing: -2.81 percent as delivered, -2.65 percent with the duplicates gone. The duplicated rows were never the story.

Revenue is the opposite. It is 32.3% with the three orders in, -2.1% with them removed, and -2.3% with them capped. A result that flips sign depending on whether three rows are included is not a result. It is a description of those three rows.

7dWhat would change the answer

The last question, and the one that stops this conversation repeating next quarter: could this test ever have detected the effect the team was hoping for?

# minimum detectable effect at 80% power, given the traffic actually collected
alpha, power = 0.05, 0.80
z_a, z_b     = stats.norm.ppf(1 - alpha / 2), stats.norm.ppf(power)

p0     = c['converted'].mean()          # baseline conversion rate
n_arm  = min(len(c), len(v))            # per-arm sample size
mde    = (z_a + z_b) * np.sqrt(2 * p0 * (1 - p0) / n_arm)

print(f'baseline {p0:.4f}   n per arm {n_arm:,}')
print(f'MDE {100*mde:.3f} pp  =  {100*mde/p0:.1f}% relative')

# and the reverse question: what would a 10% relative lift have needed
for rel in (0.05, 0.10):
    d      = p0 * rel
    needed = int(np.ceil(2 * p0 * (1 - p0) * (z_a + z_b)**2 / d**2))
    print(f'{rel:.0%} relative lift needs {needed:,} per arm')

baseline 0.0283   n per arm 19,512
MDE 0.470 pp  =  16.6% relative
5% relative lift needs 215,467 per arm
10% relative lift needs 53,867 per arm

At 1,426 visitors a day, detecting a 10 percent relative lift needs 53,867 per arm, roughly 76 days. A 5 percent lift needs 215,467 per arm, around 303 days, which is most of a year and is not a test anyone will run.

That is a genuinely useful answer even though it is not the one requested. It says the constraint is traffic, not analysis. Thornbury Home can either test larger changes, move to a metric further up the funnel where the rates are higher and the noise is lower, or accept that small conversion effects are not measurable at this volume and decide those changes on judgement instead of pretending to measure them.

Section 8Decision and Handoff

Everything above exists to support one paragraph that a non-analyst has to be able to act on.

What has to be fixed, and by whom

The specification for the rerun

1. Do not rerun until the assignment fault is confirmed fixed, verified by an A/A test whose cumulative split holds for a full week.
2. Fix the load regression first. Testing a slower page tells you about the weight, not the design.
3. Set the target effect honestly. At 1,426 visitors a day, a 10 percent relative lift needs about 76 days. If that is unacceptable, the change is too small to measure and should be decided another way.
4. Pre-register the metric definitions, including the winsorisation threshold, before traffic starts.
5. Pre-register mobile and tablet if segment answers are wanted, and power for them. Otherwise do not report them.

Reproducibility

What to take from this

• Validity checks come before effect estimates, always. The order the tests run in is the difference between catching this and shipping it.
• The suspicious arm is often the tidy one. Real traffic is lumpy. A perfectly even split across four browsers is a symptom, not a comfort.
• A daily split check misses a consistent small bias. Check the cumulative split.
• If three rows can flip your conclusion, you do not have a conclusion. Run the removal before you present, not after someone else does.
• Compute the MDE before the test, not after. Most tests that fail to reach significance were never capable of reaching it.

The uncomfortable part of this project is that the answer to the question asked is no. Not a smaller number, not a qualified yes, not a segment where it works. The value delivered is a rollout that did not happen, a load regression found, three pipeline faults documented, and a testing programme that now knows what it can and cannot measure. That is a good week’s work, and it is the job.