Product Catalogue Auto-Tagging

In the Real World · Brief · Deep learning · Advanced · 2 days

9,000 products, no attributes, and every filter on the site returning nothing. Tag the catalogue before you read the walkthrough.

The situation

Marloe Finch merged three suppliers and inherited their catalogues. 9,000 products across 20 product types, each with a title and a supplier description and nothing else. Filters are empty and search only finds what is literally in the title.

Ecommerce wants the filters working, and wants to know which ones can be trusted.

The data

What the room believes

1. One model can tag the whole catalogue.
2. A dictionary lookup is a reasonable substitute for a model.
3. Overall accuracy tells you whether the tagging is good enough.
4. Attributes the text does not state cannot be predicted.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A tagger compared against a dictionary lookup and against a trivial baseline, reported per attribute rather than as one number.
3. A measurement of how often each attribute is actually stated in the text, and what that predicts.
4. A rule for when a tag is confident enough to apply, and a defence of it.
5. A decision per attribute: which filters go live, which go live with gaps, and which should not be built.

If you want to go further

• Measure, for each attribute, how often its value literally appears in the text, then compare that to how well you predict it.
• Build the dictionary lookup first and see which attributes your model actually improves.
• Count filter values with enough products behind them, rather than reporting accuracy.
• Give each attribute its own threshold and see how differently they land.
• Find the attribute where the model barely beats random, and work out why.

In the Real World · Deep learning · Advanced · 2 days

9,000 products with no attributes, so every filter on the site is empty. One model tags all six attributes and reports 67.9 macro F1, which is an average across a 65.3 point spread. One of the six should not be shipped at all.

Section 1Problem Definition

No code yet. This gets scoped as one tagging model and it is six different problems wearing the same hat.

The problem in one sentence

An attribute can only be predicted from the text if the text contains it or implies it, and the six attributes here differ enormously in whether they do.

Business objective

Fill the filters. That means tagging enough products per filter value, accurately enough that the filter is not lying, and knowing which filters cannot be filled from the text at all.

Hypotheses

1. H1. One model can tag the whole catalogue.
2. H2. A dictionary lookup is a reasonable substitute for a model.
3. H3. Overall accuracy tells you whether the tagging is good enough.
4. H4. Attributes the text does not state cannot be predicted.

Section 2Data Collection

import numpy as np
import pandas as pd

c = pd.read_csv('data/marloe-finch-catalogue.csv')
a = pd.read_csv('data/marloe-finch-attributes.csv')
d = c.merge(a, on='sku')
d['text'] = (d['title'] + '. ' + d['description']).str.lower()
ATTRS = ['colour', 'size', 'material', 'room', 'care', 'style']

print('products      :', len(d))
print('product types :', d['product_type'].nunique())
print('title words   :', round(d['title'].str.split().str.len().mean(), 1))
print('desc words    :', round(d['description'].str.split().str.len().mean(), 1))
print('price range   :', round(d['price'].min(), 2), 'to', round(d['price'].max(), 2))

products      : 9000
product types : 20
title words   : 4.1
desc words    : 12.6
price range   : 8.01 to 319.94

Four words of title and thirteen of description. That is all there is. Any attribute not carried by seventeen words is not going to be recovered from them, and section 4d measures which ones are.

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate skus        :', int(d['sku'].duplicated().sum()))
print('missing titles        :', int(d['title'].isna().sum()))
print('missing descriptions  :', int(d['description'].isna().sum()))
print('products without every attribute :', int(d[ATTRS].isna().any(axis=1).sum()))

duplicate skus        : 0
missing titles        : 0
missing descriptions  : 0
products without every attribute : 0

3bHandling categorical mess

The product type is the one structured field that survived the merge, and it turns out to carry most of what the model knows about room and care. That is worth noticing early: some of the attributes are not being read out of the text at all, they are being inferred from what the product is.

top = d['product_type'].value_counts()
print('product types :', len(top))
print('commonest     :', top.index[0], round(100 * top.iloc[0] / len(d), 1), 'percent')
print('rarest        :', top.index[-1], round(100 * top.iloc[-1] / len(d), 1), 'percent')
print()
print('how much the type alone tells you about each attribute:')
for at in ATTRS:
    g = d.groupby('product_type')[at].agg(lambda s: s.value_counts().iloc[0] / len(s))
    print('  %-9s best single guess per type averages %5.1f percent'
          % (at, 100 * g.mean()))

product types : 20
commonest     : bath towel 5.5 percent
rarest        : bookshelf 4.4 percent

how much the type alone tells you about each attribute:
  colour    best single guess per type averages  10.5 percent
  size      best single guess per type averages  27.3 percent
  material  best single guess per type averages  43.4 percent
  room      best single guess per type averages  68.1 percent
  care      best single guess per type averages  71.2 percent
  style     best single guess per type averages  22.2 percent

The product type alone pins down care and room far better than it pins down colour, which is the first sign that these six attributes are not one problem. A cushion cover comes in any colour and is always for the living room or the bedroom.

3cDealing with outliers

Price has a long tail and is not used. It correlates with material and would leak a little signal that a new product from a new supplier will not carry in the same way.

3dHandling missing values

Nothing is missing in the file. What is missing is in the text: the attribute the supplier did not write down. That is the entire problem and it is not an imputation task.

3eHandling skewed data

Attribute values are reasonably balanced, which matters because it makes macro F1 readable and it makes a random guess a meaningful floor. Guessing at random scores between 5.8 and 24.1 depending on the attribute, and one of the six barely clears that.

3fData types and normalisation

Title and description are concatenated and lowercased for the dictionary lookup, and passed to the encoder as written. The encoder was pretrained on ordinary prose and normalising it away costs accuracy.

Section 4Exploratory Data Analysis

4aTarget variable analysis

for at in ATTRS:
    v = d[at].value_counts()
    print('%-9s %2d values   commonest %-16s %5.1f percent'
          % (at, len(v), v.index[0], 100 * v.iloc[0] / len(d)))

colour    12 values   commonest blush              8.8 percent
size       4 values   commonest small             25.6 percent
material  13 values   commonest cotton            21.9 percent
room       6 values   commonest living room       29.2 percent
care       5 values   commonest not applicable    59.1 percent
style      5 values   commonest scandinavian      20.7 percent

# Are the attributes independent, or is one a proxy for another?
import itertools
for x, y in itertools.combinations(ATTRS, 2):
    ct = pd.crosstab(d[x], d[y], normalize='index')
    best = ct.max(axis=1).mean()
    if best > 0.5:
        print('%-9s predicts %-9s at %5.1f percent' % (x, y, 100 * best))
print('pairs above 50 percent listed above, all others are below')

colour    predicts care      at  59.1 percent
size      predicts care      at  59.1 percent
material  predicts room      at  56.8 percent
material  predicts care      at  72.2 percent
room      predicts care      at  70.2 percent
pairs above 50 percent listed above, all others are below

4bNumerical variables

Only price, and it is deliberately excluded. The whole signal here is text.

4cCategorical variables

Six attributes with between four and thirteen values each. Treated as six independent single label problems rather than one large one, because the answer turns out to be different for each.

4dRelationships between variables

The measurement that predicts everything else: how often does the attribute value actually appear in the text.

SIZE_WORDS = {'small': ['small', '40cm', ' s '], 'medium': ['medium', '60cm', ' m '],
              'large': ['large', '90cm', ' l '],
              'extra large': ['extra large', '120cm', ' xl ']}

for at in ATTRS:
    if at == 'size':
        stated = np.mean([any(w in ' ' + t + ' ' for w in SIZE_WORDS[v])
                          for t, v in zip(d['text'], d[at])])
    else:
        stated = np.mean([v.lower() in t for t, v in zip(d['text'], d[at])])
    print('%-9s stated in the text %5.1f percent of the time' % (at, 100 * stated))

colour    stated in the text  88.6 percent of the time
size      stated in the text  48.7 percent of the time
material  stated in the text  60.6 percent of the time
room      stated in the text  18.4 percent of the time
care      stated in the text  14.3 percent of the time
style     stated in the text   5.5 percent of the time

Colour is written down 88.6% of the time and style 5.5%. Anything a dictionary can do is bounded by this chart. What a model adds is everything it can infer from the rest of the sentence, and section 7b measures exactly how much that is worth per attribute.

4eTesting our hypotheses

4fSubgroups

The subgroup is the attribute, and treating the six as one problem is the mistake this project exists to correct.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

5cEncoding

One classifier per attribute on a shared 384 dimension embedding. Six small models on one encode pass, which costs almost nothing and lets each attribute have its own threshold. A single multi-output model would force one threshold on all six, and section 7c is about why that would be wrong.

5dFeature selection

Nothing selected. The decision worth making is which attributes to ship, not which features to keep.

Section 6Model Selection

Section 7Model Training

7aBaselines

A 2700 product test set, 6300 for training, one classifier per attribute.

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split

E = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu').encode(
    (d['title'] + '. ' + d['description']).tolist(), batch_size=128,
    normalize_embeddings=True)
tr, te = train_test_split(np.arange(len(d)), test_size=0.30, random_state=1)
print('train', len(tr), ' test', len(te), ' embedding', E.shape)

def dictionary_tag(at):
    # Does the value literally appear. The rule anyone writes first.
    vals = sorted(d[at].unique())
    out = []
    for t in d['text']:
        pad, hit = ' ' + t + ' ', ''
        if at == 'size':
            for v in vals:
                if v in SIZE_WORDS and any(w in pad for w in SIZE_WORDS[v]):
                    hit = v
                    break
        else:
            for v in sorted(vals, key=len, reverse=True):
                if v.lower() in t:
                    hit = v
                    break
        out.append(hit)
    return np.array(out, dtype=object)

PROB = {}
print('%-9s %8s %11s %11s %10s'
      % ('attribute', 'stated', 'commonest', 'dictionary', 'model'))
for at in ATTRS:
    y = d[at].values
    dt = dictionary_tag(at)
    maj = pd.Series(y[tr]).mode()[0]
    clf = LogisticRegression(max_iter=3000, C=8.0).fit(E[tr], y[tr])
    PROB[at] = clf.predict_proba(E[te])
    f = lambda p: 100 * f1_score(y[te], p, average='macro', zero_division=0)
    print('%-9s %19.1f %11.1f %10.1f'
          % (at, f([maj] * len(te)), f(dt[te]), f(clf.predict(E[te]))))

attribute   stated   commonest  dictionary      model
colour       88.6%         1.2        86.7       89.9
size         48.7%        10.1        52.3       79.0
material     60.6%         2.7        70.8       77.2
room         18.4%         7.7        24.2       68.9
care         14.3%        14.9        34.8       67.9
style         5.5%         7.0         8.9       24.6

7bComparing candidates

The model wins on every attribute, and by wildly different margins. On colour it is worth 3.2 points, because the text already says the colour. On room it is worth 44.7, because the text almost never says the room and the model infers it from the product.

The gain is smallest on colour, which the text already states 88.6% of the time, and largest on room and care, which it almost never states. That is the shape of a model doing inference rather than pattern matching, and it is the argument for paying for one.

Style is the exception that decides the handoff. It reaches 24.6 macro F1 against 21.3 for guessing at random, a margin of 3.3 points. Style is a merchandising judgement about a product, not a property of it, and seventeen words of supplier copy do not contain it.

7cHyperparameter tuning

One dial per attribute: the confidence above which a tag is applied. Raising it trades how much of the catalogue gets tagged against how often the tag is wrong.

MIN_PER_FILTER = 25

print('%-9s %10s %10s %10s %14s'
      % ('attribute', 'threshold', 'tagged', 'accuracy', 'usable filters'))
for at in ATTRS:
    y, pr = d[at].values[te], PROB[at]
    classes = sorted(d[at].unique())
    pred = np.array(classes)[pr.argmax(1)]
    conf = pr.max(1)
    for thr in (0.0, 0.5, 0.7, 0.9):
        keep = conf >= thr
        usable = sum(1 for v in np.unique(pred[keep])
                     if (pred[keep] == v).sum() >= MIN_PER_FILTER)
        print('%-9s %10.1f %9.1f%% %9.1f %10d of %d'
              % (at, thr, 100 * keep.mean(), 100 * (pred[keep] == y[keep]).mean(),
                 usable, d[at].nunique()))

attribute  threshold     tagged   accuracy usable filters
colour           0.0     100.0%      89.9         12 of 12
colour           0.5      88.6%     100.0         12 of 12
colour           0.7      87.1%     100.0         12 of 12
colour           0.9      76.1%     100.0         12 of 12
size             0.0     100.0%      79.0          4 of 4
size             0.5      74.9%      96.9          4 of 4
size             0.7      69.3%      99.9          4 of 4
size             0.9      51.3%     100.0          4 of 4
material         0.0     100.0%      78.0         13 of 13
material         0.5      82.6%      86.7         13 of 13
material         0.7      64.8%      97.6         12 of 13
material         0.9      53.3%      99.9         10 of 13
room             0.0     100.0%      72.7          6 of 6
room             0.5      86.4%      78.4          6 of 6
room             0.7      54.4%      96.2          5 of 6
room             0.9      45.2%      99.9          5 of 6
care             0.0     100.0%      83.7          5 of 5
care             0.5      78.8%      99.0          5 of 5
care             0.7      77.3%     100.0          5 of 5
care             0.9      71.2%     100.0          5 of 5
style            0.0     100.0%      24.6          5 of 5
style            0.5       3.0%      96.3          0 of 5
style            0.7       1.1%     100.0          0 of 5
style            0.9       0.1%     100.0          0 of 5

At 0.9 confidence every attribute that gets tagged is tagged almost perfectly. The difference is how much of the catalogue clears the bar. Colour tags 76.1% and fills all 12 filters. Style tags 0.1% and fills none.

7dFinal evaluation

# What a merchandiser actually receives, per attribute, at 0.9 confidence.
for at in ATTRS:
    pr = PROB[at]
    classes = sorted(d[at].unique())
    pred, conf = np.array(classes)[pr.argmax(1)], pr.max(1)
    keep = conf >= 0.9
    usable = sum(1 for v in np.unique(pred[keep])
                 if (pred[keep] == v).sum() >= MIN_PER_FILTER)
    verdict = ('ship' if usable == d[at].nunique()
               else 'do not ship' if usable < d[at].nunique() / 2
               else 'ship with gaps flagged')
    print('%-9s tagged %5.1f%%   filters %2d of %2d   %s'
          % (at, 100 * keep.mean(), usable, d[at].nunique(), verdict))

colour    tagged  76.1%   filters 12 of 12   ship
size      tagged  51.3%   filters  4 of  4   ship
material  tagged  53.3%   filters 10 of 13   ship with gaps flagged
room      tagged  45.2%   filters  5 of  6   ship with gaps flagged
care      tagged  71.2%   filters  5 of  5   ship
style     tagged   0.1%   filters  0 of  5   do not ship

The deliverable is not a model score, it is a decision per attribute.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not report one number for a multi attribute tagger. The average here is 67.9 and the range is 65.3 points.
• Do not assume the dictionary is enough. The model beat it on all six attributes, by 44.7 points where inference was possible.
• Do not assume the dictionary is useless either. On colour it is within 3.2 points and needs no encoder.
• Do not ship a filter with three products behind it. An empty results page is worse than no filter.
• Do not use one confidence threshold across attributes. The same threshold tags 76.1% of colours and 0.1% of styles.
• Do not fine tune before checking there is signal to find. On style there is nothing in the text to fit.

Reproducibility

What to take from this

• An attribute is only predictable if the text states or implies it. Measure that before you build anything.
• A model earns its money on inference, not on reading. The biggest wins were on the attributes the text almost never states.
• Averages across attributes are meaningless. Six problems, one number, 65.3 points of spread.
• Score on what the user gets. Filters with stock behind them, not macro F1.
• Give every attribute its own threshold. They are different problems and one dial cannot serve all of them.
• Some attributes are a process problem. Style needs a form field, not a network.

The brief was nine thousand products and broken filters. Five of the six attributes come back from seventeen words of supplier copy, four of them well enough to ship untouched. The sixth was never in the text and the useful output there is not a model at all, it is one more required field on the supplier form.