Segmentation That Survives Scrutiny

In the Real World · Brief · Machine learning · Foundation · 2 to 4 hours

Marketing wants customer segments it can target next quarter, and the last set was invented in a workshop. Build them, and decide what would make them worth trusting. Work it yourself before reading the walkthrough.

The situation

Fellgate Outdoor sells outdoor gear online. You have twelve months of orders covering 1,671 customers and 2,674 orders. Marketing wants segments loaded into the email platform, and a campaign built on them will run for the next three months.

The data

What the room believes

1. The customer base contains natural groups that clustering will find.
2. The right number of segments will be obvious from an elbow in the inertia curve.
3. Segments that hold up when you rerun the model are stable enough to plan a quarter around.
4. A customer’s segment is a property of the customer.

Definition of done

1. A stated rule for choosing the number of segments, written before you see any segments.
2. A verdict on each of the four beliefs, with the evidence.
3. Segment definitions marketing could load into an email platform: a size, a profile and a sentence describing who they are.
4. Evidence that the segments are worth acting on, and a clear statement of any way in which they are not.
5. A recommendation on how often the segmentation has to be rebuilt, with the number that justifies it.

If you want to go further

• Run the clustering once without scaling the features and once with. Compare the silhouette scores, then explain the result.
• Work out what agreement two labelings of the same customers would show by pure chance, given your segment sizes, and use that as the floor for any persistence claim.
• Decide whether any of your segments is too small for the email platform to be worth targeting, and say what the threshold should be.
• Write the sentence that goes at the top of the handoff telling marketing how long the segments last.

In the Real World · Machine learning · Foundation · 2 to 4 hours

Marketing wants customer segments and the last set was invented in a workshop. K-means gives you segments in four lines. The work is proving they are worth acting on, and this set passes two tests out of three.

Section 1Problem Definition

No code yet. Clustering will always return clusters. Deciding in advance what would make them worth using is the only thing standing between a useful segmentation and a colourful one.

Business objective

Produce customer segments marketing can load into the email platform and target for the next quarter. Each segment needs a size, a description a human can act on, and a reason to believe it is real.

What would make this fail

The decision rule, written before looking

Choose the number of segments by three criteria in this order. It has to be stable under resampling, at a mean adjusted Rand index of 0.75 or better. No segment may hold fewer than 5 per cent of customers. Of whatever survives those two, take the best silhouette.

Hypotheses

1. H1. The customer base contains natural groups that clustering will find.
2. H2. The right number of segments will be obvious from an elbow in the inertia curve.
3. H3. Segments that are stable under resampling are stable enough to run a quarterly campaign against.
4. H4. A customer’s segment is a property of the customer.

Section 2Data Collection

The same three files the profit leak audit used, reduced to one row per customer.

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score, adjusted_rand_score

SEED = 20260822

lines = pd.read_csv('data/fellgate-order-lines.csv').drop_duplicates()
prod  = pd.read_csv('data/fellgate-products.csv')

d = lines.merge(prod, on='sku', validate='many_to_one')
d['dt']  = pd.to_datetime(d['order_date'])
d['rev'] = d['quantity'] * d['unit_price'] - d['line_discount']
d['returned'] = d['line_status'] == 'returned'

print('order lines :', len(d))
print('customers   :', d['customer_id'].nunique())

order lines : 4,516
customers   : 1,671

Section 3Data Preprocessing

3aDuplicates and schema checks

The duplicated export rows come out on load, as they did in the profit leak audit. The check that matters here is different: clustering needs exactly one row per customer, and a duplicate would give that customer two votes on where the centroids land.

o = d.groupby('order_id').agg(cust=('customer_id', 'first'), dt=('dt', 'first'),
                              rev=('rev', 'sum'), ret=('returned', 'sum'),
                              disc=('line_discount', 'sum'))
print('orders    :', len(o))
print('customers :', o['cust'].nunique())

orders    : 2,674
customers : 1,671

3bHandling categorical mess

Nothing to clean. Country and channel are consistent, and neither goes into the clustering: a segmentation built partly on geography will hand you geography back and call it a discovery.

3cDealing with outliers

K-means is a mean-based method, so a single extreme customer drags a centroid toward itself. Worth checking the spread before deciding whether to intervene.

AS_OF = d['dt'].max() + pd.Timedelta(days=1)

f = o.groupby('cust').agg(frequency=('rev', 'size'), monetary=('rev', 'sum'),
                          last=('dt', 'max'), disc=('disc', 'sum'))
f['recency']       = (AS_OF - f['last']).dt.days
f['aov']           = f['monetary'] / f['frequency']
f['discount_rate'] = (f['disc'] / f['monetary'].replace(0, np.nan)).fillna(0.0)

FEATURES = ['recency', 'frequency', 'monetary', 'aov', 'discount_rate']
print(f[FEATURES].describe().T[['mean', 'std', 'min', 'max']].round(2).to_string())

               mean     std    min      max
recency        123.46   98.73   1.00   365.00
frequency        1.60    0.84   1.00     6.00
monetary       232.07  182.68  11.20  1345.00
aov            145.39   95.67  11.20   709.00
discount_rate    0.01    0.04   0.00     0.25

No customer is far enough out to distort a centroid on its own. The most a single customer has spent is 1,345.00 pounds against a mean of 232.07, which is a tail rather than an outlier. Nothing is trimmed.

3dHandling missing values

None, except one that is manufactured rather than missing: discount rate is undefined for a customer who has spent nothing, and no customer has. The fillna in 3c is defensive rather than necessary, and it is left in because a regenerated dataset could produce one.

3eHandling skewed data

Monetary and average order value are right-skewed, as spend always is. Standardising in 3f handles the scale problem but not the shape, and that is a deliberate choice: a log transform would compress exactly the difference between a good customer and a great one, which is the difference marketing is paying for.

3fData types and normalisation

This is the step the whole project turns on.

print('standard deviation of each feature, in its own units')
print(f[FEATURES].std().round(2).to_string())
print()
print('widest over narrowest :',
      round(f[FEATURES].std().max() / f[FEATURES].std().min(), 1), 'to 1')

standard deviation of each feature, in its own units
recency           98.73
frequency          0.84
monetary         182.68
aov               95.67
discount_rate      0.04

widest over narrowest : 4554.5 to 1

Standard deviation of each feature before scaling. monetary is 4554.5 times wider than discount_rate, so an unscaled distance is almost entirely a monetary distance.

scaler = StandardScaler().fit(f[FEATURES])
X = scaler.transform(f[FEATURES])
print('after scaling, every feature has standard deviation', round(X.std(axis=0).mean(), 3))

after scaling, every feature has standard deviation 1.0

Section 4Exploratory Data Analysis

4aTarget variable analysis

There is no target. That is what makes this unsupervised, and it is why the evaluation in section 7 has to be built rather than looked up. What stands in for a target here is the naive run, so there is something to beat.

naive = KMeans(n_clusters=4, n_init=10, random_state=SEED).fit_predict(f[FEATURES].values)

print('sizes      :', np.bincount(naive).tolist())
print('silhouette :', round(silhouette_score(f[FEATURES].values, naive,
                                             random_state=SEED), 4))

sizes      : [669, 146, 541, 315]
silhouette : 0.367

4bNumerical variables

The five features, and why each is in.

4cCategorical variables

Deliberately excluded. Country, channel and product category are all available and none is used. Feed a category into a clustering and the clustering will return that category to you, which feels like a finding and is not one. They are far more useful afterwards, as a description of segments the numbers found on their own.

4dRelationships between variables

Frequency and monetary correlate strongly, because spending more usually means buying more often. That is not a reason to drop one. K-means has no coefficients to destabilise, and the pair together carries information neither has alone: a customer with high monetary and low frequency is a different animal from one with both high.

4eTesting our hypotheses

Two of the four can be answered before any segment is named.

4fSubgroups

The one subgroup worth checking before modelling is the single-order customer, because they have no frequency signal at all and there are a lot of them. They are kept in. A segmentation that silently excludes most of the base is not a segmentation of the base.

Section 5Feature Engineering

5aThe leakage trap

Unsupervised work has no target, so it cannot leak a label. It has a subtler version of the same fault: putting a feature in that already encodes the answer you want.

The same argument rules out anything computed after the segmentation would be used. Next quarter’s spend would separate the segments beautifully and would not exist when the campaign is built.

5bNew features

Three of the five are constructed rather than counted, and each earns its place by capturing something the raw columns cannot.

5cEncoding

Nothing to encode, because nothing categorical is in the feature set. Had country gone in, one-hot encoding it would have created five columns each with a standard deviation near 0.5, quietly outvoting the five behavioural features they sat alongside. Mixing categorical and continuous features in a distance-based method is a decision, not a default.

5dFeature selection

Five features, chosen before fitting and not tuned afterwards. Selecting features by which set produces the prettiest clusters is circular: there is no held-out truth to check the choice against, so the only thing being optimised is the appearance of the result.

Section 6Model Selection

Section 7Model Training

7aBaselines

Two baselines, and the second is the one that matters.

Any segmentation has to beat the first on usefulness and the second on persistence. It manages one of the two.

7bComparing candidates

rows = []
for k in range(2, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=SEED).fit(X)
    sil = silhouette_score(X, km.labels_, random_state=SEED)
    smallest = np.bincount(km.labels_).min() / len(X)
    rows.append((k, km.inertia_, sil, smallest))
    print(f'k={k}  inertia {km.inertia_:8.1f}  silhouette {sil:.4f}  '
          f'smallest segment {smallest:.1%}')

k=2  inertia   6122.0  silhouette 0.2984  smallest segment  33.5%
k=3  inertia   4906.9  silhouette 0.3333  smallest segment   8.4%
k=4  inertia   3994.1  silhouette 0.2992  smallest segment   7.7%
k=5  inertia   3250.6  silhouette 0.3180  smallest segment   7.6%
k=6  inertia   2842.8  silhouette 0.3111  smallest segment   7.4%
k=7  inertia   2543.5  silhouette 0.3064  smallest segment   5.8%
k=8  inertia   2346.4  silhouette 0.3102  smallest segment   1.9%

Inertia falls smoothly, as it always does, because adding a centroid can only reduce it. There is no elbow. Silhouette peaks at k=3 and every value sits between 0.2984 and 0.3333, a range too narrow to decide anything on.

7cHyperparameter tuning

The only hyperparameter that matters is k, and it is tuned on stability rather than fit.

rng = np.random.default_rng(SEED)

for k in range(2, 9):
    base = KMeans(n_clusters=k, n_init=10, random_state=SEED).fit(X)
    aris = []
    for b in range(30):
        idx = rng.choice(len(X), size=int(len(X) * 0.8), replace=False)
        alt = KMeans(n_clusters=k, n_init=10, random_state=SEED + b + 1).fit(X[idx])
        aris.append(adjusted_rand_score(base.labels_[idx], alt.labels_))
    print(f'k={k}  mean ARI {np.mean(aris):.4f}  worst {np.min(aris):.4f}')

k=2  mean ARI 0.9375  worst 0.8741
k=3  mean ARI 0.9247  worst 0.4716
k=4  mean ARI 0.9301  worst 0.3995
k=5  mean ARI 0.8992  worst 0.7264
k=6  mean ARI 0.7633  worst 0.4622
k=7  mean ARI 0.9442  worst 0.8237
k=8  mean ARI 0.8654  worst 0.6503

Silhouette never rises above 0.3333 at any k, while stability stays high almost everywhere. The two metrics are measuring different things and only one of them separates the candidates.

Stability does not discriminate either: every k from 2 to 7 clears the 0.75 threshold. Applying the rule from section 1 in order, that leaves size and then silhouette, and the answer is k=3 with a mean adjusted Rand of 0.9247.

7dFinal evaluation

What the segments are

K = 3
km = KMeans(n_clusters=K, n_init=10, random_state=SEED).fit(X)
f['segment'] = km.labels_

prof = f.groupby('segment').agg(
    customers=('frequency', 'size'), recency=('recency', 'median'),
    frequency=('frequency', 'mean'), monetary=('monetary', 'mean'),
    aov=('aov', 'mean'), discount_rate=('discount_rate', 'mean'))
prof['share']         = prof['customers'] / len(f)
prof['revenue_share'] = f.groupby('segment')['monetary'].sum() / f['monetary'].sum()
print(prof.round(3).to_string())

# do the segments actually differ, or has a distance metric just cut one cloud
for c in FEATURES:
    groups = [f.loc[f['segment'] == s, c].values for s in sorted(f['segment'].unique())]
    fstat, pv = stats.f_oneway(*groups)
    print(f'{c:15s} F {fstat:8.1f}  p {pv:.3g}')

recency         F     99.4  p below the smallest number floating point can represent
frequency       F    645.9  p below the smallest number floating point can represent
monetary        F   1145.3  p below the smallest number floating point can represent
aov             F    191.9  p below the smallest number floating point can represent
discount_rate   F   2568.6  p below the smallest number floating point can represent

The largest segment by headcount is the smallest by value. Best customers are 31.8% of customers and 59.7% of revenue.

The separation test says the segments genuinely differ on every feature, with the strongest split on discount_rate. That is necessary and nowhere near sufficient: a one-way test will find differences between any three groups a distance metric has just been asked to make as different as possible.

The test that decides it

Split the year into two equal windows, build the segmentation on the first, then ask two separate questions of the second.

def window_features(frame, as_of):
    """The same five features, computed inside one window."""
    g = frame.groupby('cust').agg(frequency=('rev', 'size'), monetary=('rev', 'sum'),
                                  last=('dt', 'max'), disc=('disc', 'sum'))
    g['recency']       = (as_of - g['last']).dt.days
    g['aov']           = g['monetary'] / g['frequency']
    g['discount_rate'] = (g['disc'] / g['monetary'].replace(0, np.nan)).fillna(0.0)
    return g[FEATURES]


mid = d['dt'].min() + (d['dt'].max() - d['dt'].min()) / 2
w1, w2 = o[o['dt'] <= mid], o[o['dt'] > mid]
f1 = window_features(w1, mid + pd.Timedelta(days=1))
f2 = window_features(w2, d['dt'].max() + pd.Timedelta(days=1))
both = sorted(set(f1.index) & set(f2.index))

sc1 = StandardScaler().fit(f1)
m1  = KMeans(n_clusters=K, n_init=10, random_state=SEED).fit(sc1.transform(f1))

# A. refit from scratch on the later window: are the DEFINITIONS reproducible
sc2 = StandardScaler().fit(f2)
m2  = KMeans(n_clusters=K, n_init=10, random_state=SEED).fit(sc2.transform(f2))
lab1  = pd.Series(m1.labels_, index=f1.index).loc[both].values
refit = pd.Series(m2.labels_, index=f2.index).loc[both].values

# B. keep the first model and score the later window: do CUSTOMERS stay put
carried = m1.predict(sc1.transform(f2.loc[both]))

print('customers in both windows :', len(both))
print('A  refit ARI              :', round(adjusted_rand_score(lab1, refit), 4))
print('B  carried ARI            :', round(adjusted_rand_score(lab1, carried), 4))
print('B  stayed in same segment :', round((lab1 == carried).mean(), 4))

customers in both windows : 380
A  refit ARI              : -0.0078
B  carried ARI            : -0.013
B  stayed in same segment : 0.3737

The same segmentation, tested three ways. Resampling within the period agrees at 0.9247. Across six months, both tests sit at zero.

Section 8Documentation and Handoff

The segment definitions, for the email platform

What to do, and who owns it

What not to do

• Do not report the silhouette as a quality score. The wrong run scored 0.367 and the right one 0.3333.
• Do not build a quarterly campaign on one snapshot. Below chance persistence at six months is the headline number here.
• Do not add more segments to make them feel more precise. k=8 puts 1.9% of customers in the smallest segment, which no one can target.
• Do not name the segments before profiling them. The names here are derived from the numbers, and they would change if the numbers did.

Reproducibility

What to take from this

• Scale before you cluster, or you are ranking on one column. A 4554.5 to 1 spread means the other features are decoration.
• A better silhouette can mean a worse segmentation. The naive run wins that metric and loses the argument.
• Stability under resampling and stability over time are different questions. This segmentation passes the first at 0.9247 and fails the second at -0.013.
• Fix the as-of date. Recency measured from today makes the segmentation unreproducible by construction.
• Decide the rule for k before you see the segments. Otherwise you are choosing the answer and calling it a method.
• A segmentation with an expiry date is still worth having. The failure here is not the clustering, it is the assumption that a customer has a segment rather than a segment this month.

Marketing asked for segments. The segments exist, they are targetable, and the most useful thing in the handoff is the sentence saying how long they last. That sentence is the difference between a campaign aimed at the right people and one aimed at where the right people were in January.