Recommendations From Order History

In the Real World · Brief · Machine learning · Core · 2 to 3 days

Five empty slots on the basket page, 30,000 orders of history, and an answer key so you can score yourself. Build the recommender before you read the walkthrough.

The situation

Ashby Kitchen sells kitchen and coffee equipment. You have 79,045 order lines across 30,000 orders from 5,932 customers, 2025-01-06 to 2026-07-05, covering 87 products.

The basket page and the post-purchase email each show five products, and today both show the same five bestsellers to everyone. Ecommerce wants a real recommendation list and a number for what it is worth.

The data

What the room believes

1. A higher hit rate means a better recommender.
2. Association rules surface the pairs worth recommending.
3. Item to item collaborative filtering beats simple rules.
4. Offline evaluation can tell you what a recommender is worth.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. At least three models compared on a holdout you can defend, including a top sellers baseline and a reorder baseline.
3. A ranked list of recommendations per product, with the rules you applied to it.
4. A statement of what is actually in your five slots, not only how often you were right.
5. A pound figure for what the list is worth, and an honest account of what that figure does and does not prove.
6. The design of the test that would settle it.

If you want to go further

• Rank the same rules on confidence and on lift, and look at what changes in the top twenty.
• Put a support floor under the rules, then sweep it, and watch what happens to your hit rate and to the kind of item you are recommending.
• Split the hit rate by whether the customer had bought the item before, and see how much of your score survives.
• Work out the opportunity cost of a slot: what is the best thing you could have put there instead of the thing you did.
• Design the live test, including how long it has to run.

In the Real World · Machine learning · Core · 2 to 3 days

Six recommenders on the same order history. The two that score best on hit rate fill three quarters of their slots with the six commonest lines in the shop. The one that scores worst is worth 11.4 times more per basket than the bestseller list the site runs today. Hit rate is not measuring what anyone is paying for.

Section 1Problem Definition

No code yet. A recommendation slot is not free. It is a fixed amount of attention on a page, and whatever goes in it displaces whatever else could have gone there. That makes this a question about opportunity cost long before it is a question about algorithms.

The problem in one sentence

Every offline recommender metric rewards predicting what the customer was going to buy anyway, and the whole commercial value of a recommendation lies in the part they were not.

Business objective

Fill five slots on the basket page with items the customer would not otherwise have bought, rank the products by how much of that they can produce, and state what has to be measured live before anyone believes a number.

Hypotheses

1. H1. A higher hit rate means a better recommender.
2. H2. Association rules surface the pairs worth recommending.
3. H3. Item to item collaborative filtering beats simple rules.
4. H4. Offline evaluation can tell you what a recommender is worth.

Section 2Data Collection

import numpy as np
import pandas as pd

lines = pd.read_csv('data/ashby-order-lines.csv', parse_dates=['order_date'])
prod = pd.read_csv('data/ashby-products.csv')
key = pd.read_csv('data/ashby-answer-key.csv')

NAME = dict(zip(prod['sku'], prod['product_name']))
SUB = dict(zip(prod['sku'], prod['subcategory']))
PRICE = dict(zip(prod['sku'], prod['unit_price']))

print('orders    :', lines['order_id'].nunique())
print('lines     :', len(lines))
print('customers :', lines['customer_id'].nunique())
print('products  :', lines['sku'].nunique(), 'of', len(prod), 'listed')
print('window    :', lines['order_date'].min().date(), 'to', lines['order_date'].max().date())

orders    : 30000
lines     : 79045
customers : 5932
products  : 87 of 87 listed
window    : 2025-01-06 to 2026-07-05

Order value across the window is 2,130,026 pounds over 78 weeks. The mean basket holds 2.63 lines and 17.1% of orders hold exactly one, which matters because a single-item order cannot be used to evaluate a basket recommender at all.

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate order and sku pairs :',
      int(lines.duplicated(subset=['order_id', 'sku']).sum()))
print('missing values                :', int(lines.isna().sum().sum()))
print('lines with a sku not in the product file :',
      int((~lines['sku'].isin(prod['sku'])).sum()))
print('answer key rows match line rows          :', len(key) == len(lines))

duplicate order and sku pairs : 0
missing values                : 0
lines with a sku not in the product file : 0
answer key rows match line rows          : True

A duplicated order and sku pair would double count a co-occurrence and inflate every rule that touches it. There are none here. On a real export there usually are, because a customer adds the same line twice and the warehouse system splits it.

3bHandling categorical mess

Two categorical fields do real work later. The subcategory identifies substitutes: two products in Pans compete for the same purchase, so recommending one when the other is already in the basket is not a recommendation. The consumable flag identifies replenishment, which is a different job with a different page.

3cDealing with outliers

size = lines.groupby('order_id').size()
print('basket size: mean %.2f  median %d  p99 %d  max %d'
      % (size.mean(), size.median(), size.quantile(0.99), size.max()))
print('single item orders :', round(100 * (size == 1).mean(), 1), 'percent')
print('usable for evaluation :', int((size > 1).sum()), 'orders')

basket size: mean 2.63  median 2  p99 6  max 9
single item orders : 17.1 percent
usable for evaluation : 24876 orders

No basket is large enough to distort a co-occurrence count. The outlier that matters in this kind of data is the trade customer with forty lines in one order, because every pair in that basket gets counted and one buyer can invent a rule on their own. There is none here. Look for one before you trust any rule.

3dHandling missing values

Nothing is missing. The absence that matters in a recommender is not a null, it is a product with too few orders to say anything about. That is handled with a support floor in section 5, not with imputation.

3eHandling skewed data

pen = lines.groupby('sku')['order_id'].nunique() / lines['order_id'].nunique()
print('top product penetration :', round(100 * pen.max(), 1), 'percent')
print('top 10 products are     :', round(100 * lines['sku'].isin(
    pen.nlargest(10).index).mean(), 1), 'percent of all lines')
print('products in under 1 percent of orders :', int((pen < 0.01).sum()))

top product penetration : 32.6 percent
top 10 products are     : 53.7 percent of all lines
products in under 1 percent of orders : 37

This skew is the whole difficulty. A handful of lines appear in a third of baskets, so any model that predicts them will look accurate, and any metric that counts hits will reward it for doing so.

3fData types and normalisation

baskets = lines.groupby('order_id')['sku'].apply(lambda s: list(dict.fromkeys(s)))
odate = lines.groupby('order_id')['order_date'].min()
ocust = lines.groupby('order_id')['customer_id'].first()
print('baskets built :', len(baskets))
print('mean distinct items per basket :', round(baskets.apply(len).mean(), 2))

baskets built : 30000
mean distinct items per basket : 2.63

Quantity is deliberately dropped. A basket is a set of products for this purpose: buying three packs of filters is one co-occurrence with the machine, not three.

Section 4Exploratory Data Analysis

4aTarget variable analysis

There is no target column. The target is constructed: hide one item from a basket, show the model the rest, and ask whether the hidden item appears in the five it recommends. That is hit rate at five, and by the end of section 7 the interesting question will be what it does not count rather than what it does.

4bNumerical variables

top = (lines.groupby('sku')['order_id'].nunique() / lines['order_id'].nunique()).nlargest(6)
for sku, p in top.items():
    print('%-8s %-32s %5.1f percent of orders' % (sku, NAME[sku], 100 * p))

AK-803   Surface spray, 750ml              32.6 percent of orders
AK-804   Microfibre cloths, 10 pack        30.4 percent of orders
AK-802   Dishwasher tablets, 60            17.6 percent of orders
AK-124   Single origin beans, 500g         10.3 percent of orders
AK-908   Silicone spatula set               9.9 percent of orders
AK-312   Muffin tray, 12 hole               8.6 percent of orders

Three products sit far above everything else. Surface spray, 750ml is in 32.6% of orders and Microfibre cloths, 10 pack in 30.4%. They are the kitchen equivalent of a carrier bag: everyone takes one, and no recommendation caused it.

4cCategorical variables

This is where the answer key earns its place. Every line in the file arrived in the basket for one of four reasons, and only one of them is a recommendation opportunity.

print(key['origin'].value_counts(normalize=True).mul(100).round(1))

origin
base          45.4
seed          32.3
repeat        16.8
complement     5.5
Name: proportion, dtype: float64

Seed is what the customer came for. Popularity is the everyday lines going in on the way past. Repeat is a replenishment of something they already buy. Complement is the item that would not be in the basket without something else in it, and it is 5.5% of all lines. That thin slice is the whole prize.

4dRelationships between variables

Association rules come with three numbers and most write-ups rank on the wrong one. Support is how often the pair appears. Confidence is how often the companion appears given the anchor. Lift is confidence divided by the companion’s own base rate.

def rules_from(bk, min_pairs=1):
    n_bk, cnt, item = len(bk), {}, {}
    for b in bk:
        u = sorted(set(b))
        for s in u:
            item[s] = item.get(s, 0) + 1
        for i, a in enumerate(u):
            for c in u[i + 1:]:
                cnt[(a, c)] = cnt.get((a, c), 0) + 1
    rows = []
    for (a, c), k in cnt.items():
        if k < min_pairs:
            continue
        for x, y in ((a, c), (c, a)):
            rows.append({'anchor': x, 'companion': y, 'pairs': k, 'support': k / n_bk,
                         'confidence': k / item[x],
                         'lift': (k / n_bk) / ((item[x] / n_bk) * (item[y] / n_bk))})
    return pd.DataFrame(rows), item, n_bk

rl, item_cnt, n_bk = rules_from(baskets)
print('candidate pairs :', len(rl) // 2)
print(rl.nlargest(8, 'confidence')[['anchor', 'companion', 'pairs',
                                    'confidence', 'lift']].round(3).to_string(index=False))

candidate pairs : 3111
anchor companion  pairs  confidence   lift
AK-906    AK-301    121       0.688 16.343
AK-323    AK-301    233       0.465 11.056
AK-321    AK-312    675       0.448  5.225
AK-514    AK-503     91       0.412 10.268
AK-412    AK-402     25       0.391 65.836
AK-413    AK-401     86       0.374 14.022
AK-128    AK-803     20       0.370  1.137
AK-502    AK-803     35       0.347  1.063

The first six are genuine. From rank 7 down, all 14 remaining rules in the top twenty point at just 2 products, and the highest lift among them is 1.14. Confidence has found that surface spray, 750ml is popular. It rediscovers that fact once for every anchor in the catalogue.

truth = pd.read_csv('data/ashby-true-complements.csv')
TRUE_PAIRS = set(zip(truth['anchor_sku'], truth['companion_sku']))
rl['true_pair'] = [(a, c) in TRUE_PAIRS or (c, a) in TRUE_PAIRS
                   for a, c in zip(rl['anchor'], rl['companion'])]
FLOOR = 60
strong = rl[rl['pairs'] >= FLOOR]

for label, df in [('confidence', rl.nlargest(20, 'confidence')),
                  ('lift, no support floor', rl.nlargest(20, 'lift')),
                  ('lift, 60 basket floor', strong.nlargest(20, 'lift'))]:
    print('%-24s real pairs in top 20: %4.1f percent   median baskets behind a rule: %4d'
          % (label, 100 * df['true_pair'].mean(), df['pairs'].median()))

confidence               real pairs in top 20: 30.0 percent   median baskets behind a rule:   63
lift, no support floor   real pairs in top 20: 80.0 percent   median baskets behind a rule:   69
lift, 60 basket floor    real pairs in top 20: 100.0 percent   median baskets behind a rule:  124

Ranking on confidence puts 70.0% of the top twenty on an everyday line. Ranking on lift fixes that and introduces a different problem, which is rules built on a handful of baskets. A floor of 60 baskets under the pair removes both failures and recovers 50.0% of the 40 real links in twenty rules.

4eTesting our hypotheses

4fSubgroups

lines = lines.sort_values(['customer_id', 'order_date', 'order_id'])
seen, flag = set(), []
for c, s in zip(lines['customer_id'], lines['sku']):
    flag.append((c, s) in seen)
    seen.add((c, s))
lines['seen_before'] = flag
print('lines the customer had bought before :',
      round(100 * lines['seen_before'].mean(), 1), 'percent')
print('orders containing at least one       :',
      round(100 * lines.groupby('order_id')['seen_before'].any().mean(), 1), 'percent')

lines the customer had bought before : 33.8 percent
orders containing at least one       : 62.5 percent

62.5% of orders contain something the customer has bought before. A model that does nothing but list a customer’s own history will therefore look competent, and section 7a uses exactly that as a baseline because it is the honest floor for this problem.

Section 5Feature Engineering

5aThe leakage trap

There are three ways to leak in a recommender and the first two are easy to do by accident.

5bNew features

5cEncoding

The customer by item matrix is binary, not counts. Someone who orders filters twelve times a year is not twelve times more similar to another filter buyer, and counts let the heavy repeat buyers dominate every similarity in the matrix.

5dFeature selection

The only real tuning decision in this project is the support floor, and section 7c shows it is not a modelling parameter at all. It is a dial that trades hit rate against commercial value, in that direction, every time.

Section 6Model Selection

Six candidates, each of which is a real production option that some shop is running right now.

Section 7Model Training

7aBaselines

cutoff = odate.max() - pd.Timedelta(weeks=12)
train_ids = list(odate.index[odate <= cutoff])
after = list(odate.index[odate > cutoff])
test_ids = [o for o in after if len(baskets[o]) > 1]
print('cutoff          :', cutoff.date())
print('train orders    :', len(train_ids))
print('orders after it :', len(after))
print('usable for test :', len(test_ids), 'with two or more lines')

cutoff          : 2026-04-12
train orders    : 24,980
orders after it : 5,020
usable for test : 4,231 with two or more lines

Twelve weeks held out at the end, never touched during rule mining. Every model below sees the same 4,231 baskets and the same hidden item in each one, drawn once, so the comparison is like for like.

7bComparing candidates

# Everything a model needs, built from training orders only. Called again in 7c
# with a different floor and with a different split, so it is a function.
def fit(train_ids, floor=FLOOR):
    global pop, conf, lift, liftf, cos, hist, rr
    rr, ic, nb = rules_from(baskets.loc[train_ids])
    rr['same_subcat'] = [SUB[a] == SUB[c] for a, c in zip(rr['anchor'], rr['companion'])]
    pop = (pd.Series(ic) / nb).sort_values(ascending=False)

    conf, lift, liftf = {}, {}, {}
    for r in rr.itertuples():
        conf.setdefault(r.anchor, []).append((r.companion, r.confidence))
        lift.setdefault(r.anchor, []).append((r.companion, r.lift))
        if r.pairs >= floor and not r.same_subcat:
            liftf.setdefault(r.anchor, []).append((r.companion, r.lift))

    tl = lines[lines['order_id'].isin(set(train_ids))]
    cols = sorted(tl['sku'].unique())
    mat = pd.crosstab(tl['customer_id'], tl['sku']).clip(upper=1).astype(float).values
    nrm = np.linalg.norm(mat, axis=0)
    nrm[nrm == 0] = 1
    S = (mat.T @ mat) / np.outer(nrm, nrm)
    np.fill_diagonal(S, 0.0)
    cos = pd.DataFrame(S, index=cols, columns=cols)
    hist = tl.groupby('customer_id')['sku'].apply(list).to_dict()

fit(train_ids)

def recommend(model, visible, cust, k=5):
    own, block = set(hist.get(cust, [])), set(visible)
    if model == 'popular':
        return [s for s in pop.index if s not in block][:k]
    if model == 'repeat':
        prev = [s for s in pd.Series(hist.get(cust, [])).value_counts().index
                if s not in block]
        pad = [s for s in pop.index if s not in block and s not in prev]
        return (prev + pad)[:k]
    if model == 'cosine':
        here = [s for s in visible if s in cos.index]
        if not here:
            return [s for s in pop.index if s not in block][:k]
        sc = cos.loc[here].sum().drop(labels=[s for s in block if s in cos.index],
                                      errors='ignore')
        return list(sc.nlargest(k).index)
    src = {'confidence': conf, 'lift': lift, 'complement': liftf}[model]
    sc = {}
    for s in visible:
        for c, val in src.get(s, []):
            if c in block or (model == 'complement' and c in own):
                continue
            sc[c] = max(sc.get(c, 0.0), val)
    out = [c for c, _ in sorted(sc.items(), key=lambda x: -x[1])][:k]
    return out + [s for s in pop.index
                  if s not in block and s not in out][:k - len(out)]

# One hidden item per test basket, drawn once so every model answers the same question.
hr = np.random.default_rng(11)
HELD = {o: baskets[o][int(hr.integers(0, len(baskets[o])))]
        for o in baskets.index if len(baskets[o]) > 1}

MODELS = [('popular', 'Top sellers'), ('repeat', 'What they bought before'),
          ('confidence', 'Association rules, confidence'),
          ('lift', 'Association rules, lift'), ('cosine', 'Item to item, cosine'),
          ('complement', 'Complement rules, filtered')]
BEST6 = list(pen.nlargest(6).index)

def evaluate(ids):
    res = {m: dict(n=0, hit=0, new=0, comp=0, value=0.0, slots=0, owned=0, best=0, disc=0)
           for m, _ in MODELS}
    for oid in ids:
        b = baskets[oid]
        held = HELD[oid]
        vis = [s for s in b if s != held]
        cust, own = ocust[oid], set(hist.get(ocust[oid], []))
        for m, _ in MODELS:
            recs = recommend(m, vis, cust)
            r = res[m]
            r['n'] += 1
            r['slots'] += len(recs)
            r['owned'] += sum(1 for c in recs if c in own)
            r['best'] += sum(1 for c in recs if c in BEST6)
            r['disc'] += sum(1 for c in recs if c not in own and c not in BEST6)
            if held in recs:
                r['hit'] += 1
                r['new'] += held not in own
                if ORIGIN.get((oid, held)) == 'complement':
                    r['comp'] += 1
                    r['value'] += PRICE[held]
    return pd.DataFrame([
        {'model': label, 'hit': 100 * r['hit'] / r['n'], 'new_hit': 100 * r['new'] / r['n'],
         'complement_hit': 100 * r['comp'] / r['n'],
         'value_per_1000': 1000 * r['value'] / r['n'],
         'owned_slots': 100 * r['owned'] / r['slots'],
         'top6_slots': 100 * r['best'] / r['slots'],
         'discovery_slots': 100 * r['disc'] / r['slots']}
        for (m, label), r in [((m, l), res[m]) for m, l in MODELS]])

ORIGIN = {(r.order_id, r.sku): r.origin for r in key.itertuples()}
out = evaluate(test_ids)
print(out[['model', 'hit', 'new_hit']].round(1).to_string(index=False))

                        model  hit  new_hit
                  Top sellers 43.3     14.4
      What they bought before 37.4      5.1
Association rules, confidence 49.0     18.8
      Association rules, lift  9.9      7.3
         Item to item, cosine 45.8     16.7
   Complement rules, filtered 17.9     13.0

On the metric every recommender paper reports, the ranking is clear. Association rules, confidence wins at 49.0%, cosine is close behind, and the filtered complement model looks like a failure at 17.9%. Ranking on lift looks like a catastrophe at 9.9%.

The second column is the first crack. New-item hit rate counts only the hits on items the customer had never bought before. What they bought before scores 37.4% on hit rate and 5.1% on new items, because almost everything it gets right is something the customer already had.

7cHyperparameter tuning

There is one dial. The support floor decides how many baskets must sit behind a pair before it is allowed to become a rule.

for floor in (5, 20, 60, 150, 400):
    fit(train_ids, floor)
    row = evaluate(test_ids).set_index('model').loc['Complement rules, filtered']
    print('floor %3d   hit %5.1f   value per 1000 %7.2f   discovery slots %5.1f'
          % (floor, row['hit'], row['value_per_1000'], row['discovery_slots']))

fit(train_ids)          # back to the chosen floor before anything else runs

floor   5   hit   8.6   value per 1000  472.51   discovery slots  98.5
floor  20   hit  11.4   value per 1000  482.44   discovery slots  92.7
floor  60   hit  17.9   value per 1000  453.13   discovery slots  79.9
floor 150   hit  28.6   value per 1000  337.25   discovery slots  58.0
floor 400   hit  39.4   value per 1000   85.46   discovery slots  21.6

Raise the floor and the hit rate climbs from 8.6% to 39.4%, because a higher floor admits only common products and common products are easy to predict. Over the same range the commercial value falls from 472.51 pounds to 85.46 pounds per thousand baskets. The dial does not trade accuracy against complexity. It trades a metric against money.

The second thing worth checking is the split. A random split of rows rather than a split by date is the standard shortcut, and the standard warning is that it leaks the future.

# The same evaluation with a random share of orders held out instead of the
# final twelve weeks. Rules, history and similarities are rebuilt on the rest.
share = len(after) / len(odate)
mask = pd.Series(np.random.default_rng(7).random(len(odate)) < share, index=odate.index)
fit(list(odate.index[~mask]))
rand = evaluate([o for o in odate.index[mask] if len(baskets[o]) > 1])
fit(train_ids)

gap = out[['model', 'hit']].merge(rand[['model', 'hit']], on='model',
                                  suffixes=('_time', '_random'))
gap['gap'] = gap['hit_random'] - gap['hit_time']
for r in gap.itertuples():
    print('%-32s time %5.1f   random %5.1f   gap %+5.1f'
          % (r.model, r.hit_time, r.hit_random, r.gap))

Top sellers                      time  43.3   random  41.9   gap  -1.4
What they bought before          time  37.4   random  40.5   gap  +3.1
Association rules, confidence    time  49.0   random  48.6   gap  -0.4
Association rules, lift          time   9.9   random  12.0   gap  +2.1
Item to item, cosine             time  45.8   random  45.4   gap  -0.4
Complement rules, filtered       time  17.9   random  18.0   gap  +0.1

7dFinal evaluation

Three columns decide this, and none of them is the hit rate. What is in the slots, how many of the hits were on items the customer would have found anyway, and what the rest are worth.

print(out[['model', 'hit', 'owned_slots', 'top6_slots', 'discovery_slots',
           'complement_hit', 'value_per_1000']].round(2).to_string(index=False))

                        model   hit  owned_slots  top6_slots  discovery_slots  complement_hit  value_per_1000
                  Top sellers 43.35        48.94       96.91             1.85            0.35           49.63
      What they bought before 37.39        86.41       46.45             0.20            0.43           61.31
Association rules, confidence 49.00        45.54       73.93            19.07            3.78          540.20
      Association rules, lift  9.86         7.15        1.20            92.19            4.09          563.77
         Item to item, cosine 45.83        48.74       76.40            15.22            1.06          139.12
   Complement rules, filtered 17.87         7.02       20.10            79.90            3.31          453.13

Top sellers puts 96.9% of its slots on the six most common lines and cosine puts 76.4%. What they bought before puts 86.4% of its slots on things the customer already owns. These are the models with the best hit rates, and this is why.

Same models, same holdout, same hidden items. The ordering is close to reversed. Association rules, lift reaches 563.77 pounds while scoring 9.9% on hit rate. Top sellers reaches 49.63 pounds while scoring 43.3%.

Across the six models the rank correlation between hit rate and value is -0.257. Not zero, which would at least be honest noise. Negative, which means that on this data choosing the model with the better hit rate is slightly worse than choosing at random.

The tallest value bar belongs to the shortest hit rate bar. Cosine is second on hit rate and fourth on value. Top sellers is third on hit rate and last on value. Only association rules on confidence does well on both, and section 7d explains why that is an accident of this catalogue rather than a reason to rank on hit rate. Value is plotted in tens of pounds so the two series share an axis, which changes neither ordering.

At 400,000 baskets a year, the spread between the best and worst model on this measure is 225,508 pounds against 19,852 pounds of complement revenue reached. Choosing on hit rate picks association rules on confidence, worth 216,080 pounds, which is close to the best available and is luck rather than method. One place down the same leaderboard sits cosine, 3.2 points behind on the metric and 160,432 pounds behind on the money. Two places down sits the bestseller list the site already runs, 19,852 pounds. Nothing in the hit rate column tells you which of those three you are choosing.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not choose a model on hit rate. Across these six the rank correlation between hit rate and value is -0.257.
• Do not let the reorder list masquerade as a recommender. What they bought before scores 37.4% with 86.4% of its slots on items the customer already owns.
• Do not rank association rules on confidence. 14 of the top twenty here point at 2 products with a lift of about one.
• Do not mine rules on the full file. Mine on training orders only, or every rule already knows the basket it is scored on.
• Do not raise the support floor to make the metric look better. It works, and it costs 387.05 pounds per thousand baskets to do it.
• Do not reach for a bigger model first. At 87 products the evaluation is the problem, not the architecture.

Reproducibility

What to take from this

• A recommendation slot is only worth the purchases it creates. Every offline metric measures purchases that happened, which is a different set.
• Hit rate rewards popularity and replenishment. Both are things the customer would have reached without you, and together they are most of what a good hit rate is made of.
• Rank rules on lift and put a floor under them. Confidence finds your bestsellers, bare lift finds coincidences, and the fix is one line.
• Report what is in the slots, not just what was hit. Share of slots on top sellers and on already owned items make a hit rate readable.
• Exclude substitutes and owned items before you ship. Both look like hits offline and neither is a recommendation.
• Only a holdout can price a recommender. Everything before that is a shortlist of things worth testing.

The basket page will show five things whatever happens. The question was never which model predicts best, it was what those five slots are for, and the answer is the small share of purchases that would not exist without them. On this file that share is 5.5% of all order lines, and every metric in general use is dominated by the other 94.5%.