Review Themes at Scale

In the Real World · Brief · Deep learning · Core · 2 days

12,000 reviews in six months and nobody has read more than a sample. Build the monthly theme report before you read the walkthrough.

The situation

Danecroft Home sells homeware. 12,000 customer reviews arrived between 2026-01 and 2026-06. Each month somebody reads about fifty of them and writes a summary, and that summary is the only view anyone has of what customers are saying.

The customer team wants a report they can act on, and an answer to which problem to fix first.

The data

What the room believes

1. Topic modelling will find the themes in the reviews.
2. Sentence embeddings and clustering will find them properly.
3. Hand labelling is too slow to be worth considering.
4. An off the shelf sentiment model can be trusted on our text.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. At least one unsupervised and one supervised approach, compared on how many of the ten themes each actually recovers.
3. A monthly report with volume, sentiment and trend for every theme.
4. A recommendation for which theme to fix first, and a defence of the ranking you chose over the other ones available.
5. The cost of your approach, including any human time it needs.

If you want to go further

• Run a topic model, then check each topic against the theme list by hand and count how many map cleanly.
• Sweep the number of clusters and watch what happens to your quality metric. Then decide whether that metric could ever choose k for you.
• Label two or three hundred reviews yourself and see where that lands against the unsupervised result.
• Rank the themes by mentions, by negative mentions and by growth, and see whether the three lists agree.
• Plot each theme by month rather than in total, and look for the one that changed.

In the Real World · Deep learning · Core · 2 days

12,000 reviews in six months and somebody reads a sample of fifty. Topic modelling recovers 4 of the 10 themes actually in the file and misses the one that tripled. An afternoon of hand labelling beats it outright.

Section 1Problem Definition

No code yet. Everybody asks for this project as topic modelling, and topic modelling is the part of it that does not work.

The problem in one sentence

Reading a sample tells you what is common, and what is common is almost never what is worth fixing.

Business objective

Produce a monthly report the customer team can act on, and say which theme to fix first, with the evidence for choosing it over the louder ones.

Hypotheses

1. H1. Topic modelling will find the themes in the reviews.
2. H2. Sentence embeddings and clustering will find them properly.
3. H3. Hand labelling is too slow to be worth considering.
4. H4. An off the shelf sentiment model can be trusted on our text.

Section 2Data Collection

import numpy as np
import pandas as pd

r = pd.read_csv('data/danecroft-reviews.csv')
key = pd.read_csv('data/danecroft-answer-key.csv')
d = r.merge(key, on='review_id')
d['month'] = d['review_date'].str[:7]
d['theme_list'] = d['themes'].str.split('|')

print('reviews    :', len(d))
print('months     :', d['month'].nunique())
print('mean stars :', round(d['stars'].mean(), 2))
print('mean words :', round(d['review_text'].str.split().str.len().mean(), 1))
print('one theme  :', round(100 * (d['n_themes'] == 1).mean(), 1), 'percent')
print('mentions   :', int(d['n_themes'].sum()))

reviews    : 12000
months     : 6
mean stars : 3.32
mean words : 17.6
one theme  : 62.4 percent
mentions   : 16511

Short reviews, a mean of 17.6 words. 62.4% raise one theme and the rest raise two, which is why everything below is multi-label rather than a single choice per review.

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate review ids :', int(d['review_id'].duplicated().sum()))
print('missing text         :', int(d['review_text'].isna().sum()))
print('stars outside 1 to 5 :', int((~d['stars'].between(1, 5)).sum()))
print('key rows match       :', len(d) == len(r))

duplicate review ids : 0
missing text         : 0
stars outside 1 to 5 : 0
key rows match       : True

3bHandling categorical mess

The star rating is the only categorical field and it is deliberately not used as a label. A one star review can be furious about one theme and complimentary about another, and collapsing that to a single number is the thing this project exists to replace.

3cDealing with outliers

No length outliers worth trimming. The longest reviews raise two themes, which is signal rather than noise.

3dHandling missing values

Nothing missing. The absence that matters is a theme nobody happened to write about this month, and that is handled by reading volumes month by month rather than by imputing anything.

3eHandling skewed data

Theme volume is heavily skewed, which is the first trap. The largest theme is 4.0 times the smallest, so any method that optimises overall fit will spend its capacity on the biggest theme and quietly drop the small ones.

3fData types and normalisation

Two representations are built. A bag of words with English stopwords removed, for the topic model, because that is what it expects. And raw text for the sentence encoder, because it was pretrained on ordinary writing and stripping it back costs accuracy.

Section 4Exploratory Data Analysis

4aTarget variable analysis

from collections import Counter
c = Counter(t for row in d['theme_list'] for t in row)
for t, n_t in c.most_common():
    print('%-22s %5d  %5.1f percent of reviews' % (t, n_t, 100 * n_t / len(d)))

delivery_speed          3488   29.1 percent of reviews
product_quality         2385   19.9 percent of reviews
sizing_fit              1869   15.6 percent of reviews
pricing_value           1657   13.8 percent of reviews
customer_service        1555   13.0 percent of reviews
delivery_damage         1270   10.6 percent of reviews
returns_process         1210   10.1 percent of reviews
description_accuracy    1174    9.8 percent of reviews
website_checkout        1025    8.5 percent of reviews
stock_availability       878    7.3 percent of reviews

Delivery speed is mentioned in 29.1% of reviews, more than any other theme. It is also 20.5% negative, which is the lowest rate in the file. People talk about delivery because delivery is what happens to them, not because it is going wrong.

4bNumerical variables

Star ratings correlate with the sentiment in the text at 0.366, which is weaker than most people expect and is the reason stars cannot stand in for theme level sentiment.

4cCategorical variables

Each review carries one or two themes and a sentiment on each. That combination is what makes the three different rankings in section 7d possible, and it is the thing a star rating throws away.

4dRelationships between variables

neg = {}
for row, sents in zip(d['theme_list'], d['sentiments'].str.split('|')):
    for t, s in zip(row, sents):
        neg.setdefault(t, []).append(1 if s == 'negative' else 0)
for t in sorted(neg, key=lambda x: -np.mean(neg[x])):
    print('%-22s %5.1f percent negative, %5d mentions'
          % (t, 100 * np.mean(neg[t]), len(neg[t])))

sizing_fit              89.7 percent negative,  1869 mentions
delivery_damage         88.3 percent negative,  1270 mentions
returns_process         62.6 percent negative,  1210 mentions
stock_availability      59.3 percent negative,   878 mentions
customer_service        55.3 percent negative,  1555 mentions
description_accuracy    52.0 percent negative,  1174 mentions
website_checkout        48.8 percent negative,  1025 mentions
product_quality         41.3 percent negative,  2385 mentions
pricing_value           38.1 percent negative,  1657 mentions
delivery_speed          20.5 percent negative,  3488 mentions

The two series barely relate. Delivery speed is the biggest and the least negative. Sizing and fit is 89.7% negative on 15.6% of reviews. Ranking themes by how often they come up puts the calmest theme first.

4eTesting our hypotheses

4fSubgroups

The interesting subgroup is time. Four of the six months look alike and two do not, and section 7d is about the theme responsible.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import CountVectorizer

THEMES = pd.read_csv('data/danecroft-theme-list.csv')['theme_key'].tolist()
Y = np.zeros((len(d), len(THEMES)), dtype=int)
NEGM = np.zeros((len(d), len(THEMES)), dtype=int)
for i, (ths, sents) in enumerate(zip(d['theme_list'], d['sentiments'].str.split('|'))):
    for t, s in zip(ths, sents):
        Y[i, THEMES.index(t)] = 1
        if s == 'negative':
            NEGM[i, THEMES.index(t)] = 1

cv = CountVectorizer(max_df=0.5, min_df=5, stop_words='english')
X = cv.fit_transform(d['review_text'])
E = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu').encode(
    d['review_text'].tolist(), batch_size=128, normalize_embeddings=True)
print('bag of words :', X.shape)
print('embeddings   :', E.shape)

5cEncoding

Themes are encoded as a binary matrix, one column per theme, because 37.6% of reviews raise two. Forcing one label per review would discard a third of the mentions in the file.

5dFeature selection

None. The choice that matters here is not which features to keep, it is whether to spend an afternoon labelling, and section 7b prices it.

Section 6Model Selection

Section 7Model Training

7aBaselines

Two unsupervised methods, each given exactly 10 groups, which is already more help than they would get in real life. Each discovered group is matched to the planted theme it mostly contains, and scored on how pure that match is and how many of the real themes get found at all.

from sklearn.cluster import KMeans
from sklearn.decomposition import LatentDirichletAllocation

solo = d['n_themes'] == 1
solo_theme = np.array([THEMES.index(t[0]) if len(t) == 1 else -1
                       for t in d['theme_list']])

def align(assign, k):
    # Match every discovered group to the planted theme it mostly contains.
    mask = solo.values & (assign >= 0)
    rows = []
    for g in range(k):
        m = mask & (assign == g)
        if m.sum() == 0:
            continue
        vals, counts = np.unique(solo_theme[m], return_counts=True)
        rows.append((int(m.sum()), THEMES[int(vals[counts.argmax()])],
                     counts.max() / m.sum()))
    covered = {r[1] for r in rows}
    purity = sum(r[2] * r[0] for r in rows) / max(sum(r[0] for r in rows), 1)
    return purity, len(covered), sorted(set(THEMES) - covered)

K = len(THEMES)
lda = LatentDirichletAllocation(n_components=K, random_state=0,
                                learning_method='batch', max_iter=25).fit(X)
p_lda, n_lda, miss_lda = align(lda.transform(X).argmax(1), K)
km = KMeans(n_clusters=K, n_init=10, random_state=0).fit(E)
p_km, n_km, miss_km = align(km.labels_, K)

print('LDA      purity %.3f  recovered %d of %d' % (p_lda, n_lda, K))
print('  missed:', ', '.join(miss_lda))
print('KMeans   purity %.3f  recovered %d of %d' % (p_km, n_km, K))
print('  missed:', ', '.join(miss_km))

LDA      purity 0.3981  recovered 4 of 10
  missed: Customer service, Damage and packaging, Description accuracy, Returns process, Stock availability, Website and checkout
KMeans   purity 0.4724  recovered 7 of 10
  missed: Returns process, Stock availability, Website and checkout

LDA finds 4 of 10 and misses 6, including damage and packaging. That omission is not a detail: section 7d shows it is the theme the business most needed to see.

terms = np.array(cv.get_feature_names_out())
for i in range(K):
    print('%2d  %s' % (i + 1, ', '.join(terms[lda.components_[i].argsort()[-6:][::-1]])))

Read those topics as a person would. Several are recognisable, several are the same theme split in two, and several are a blur of words that co-occur without meaning anything together. This is what a topic model produces on short reviews, and it is why the output of one so often ends up being interpreted rather than used.

7bComparing candidates

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

tr_all, te = train_test_split(np.arange(len(d)), test_size=0.30, random_state=1)
for n_lab in (100, 300, 600, 1200, len(tr_all)):
    tr = tr_all[:n_lab]
    pred = np.zeros((len(te), len(THEMES)), dtype=int)
    for j in range(len(THEMES)):
        if Y[tr, j].sum() < 5 or Y[tr, j].sum() == len(tr):
            continue
        pred[:, j] = LogisticRegression(max_iter=2000, C=4.0,
                                        class_weight='balanced').fit(
            E[tr], Y[tr, j]).predict(E[te])
    print('%5d labels  micro F1 %5.1f  macro F1 %5.1f  precision %5.1f  recall %5.1f'
          % (n_lab,
             100 * f1_score(Y[te], pred, average='micro', zero_division=0),
             100 * f1_score(Y[te], pred, average='macro', zero_division=0),
             100 * precision_score(Y[te], pred, average='micro', zero_division=0),
             100 * recall_score(Y[te], pred, average='micro', zero_division=0)))

  100 labels  micro F1  81.8  macro F1  80.1  precision  85.7  recall  78.2
  300 labels  micro F1  91.7  macro F1  90.8  precision  93.5  recall  90.0
  600 labels  micro F1  95.8  macro F1  95.4  precision  97.7  recall  93.9
 1200 labels  micro F1  97.1  macro F1  97.0  precision  98.8  recall  95.5
 8400 labels  micro F1  99.1  macro F1  99.0  precision  99.4  recall  98.8

81.8% micro F1 from 100 labels and 91.7% from 300. At half a minute a review, 300 is 2.5 hours of somebody’s time, or 52.50 pounds. Labelling the whole corpus would reach 99.1% and take about a fortnight, which is the wrong trade.

7cTuning

The unsupervised methods have one parameter and it is the one nobody can set honestly: how many groups there are.

for k in (5, 8, 10, 14, 20):
    p_k, n_k, _ = align(KMeans(n_clusters=k, n_init=10, random_state=0).fit(E).labels_, k)
    print('k = %2d   purity %.3f   themes recovered %d' % (k, p_k, n_k))

k =  5   purity 0.428   themes recovered 3
k =  8   purity 0.449   themes recovered 4
k = 10   purity 0.472   themes recovered 7
k = 14   purity 0.488   themes recovered 7
k = 20   purity 0.581   themes recovered 8

Purity rises monotonically with k, from 0.4282 at 5 clusters to 0.5805 at 20, and would keep rising until every review was its own cluster. Even at 20 it recovers only 8 of the 10 real themes. There is no value of k that produces the list the business already works from.

The sentiment model, tested rather than trusted

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = 'distilbert-base-uncased-finetuned-sst-2-english'
tok = AutoTokenizer.from_pretrained(name)
mdl = AutoModelForSequenceClassification.from_pretrained(name).eval()
scores = []
with torch.no_grad():
    for i in range(0, len(d), 64):
        b = tok(d['review_text'].iloc[i:i + 64].tolist(), truncation=True,
                max_length=96, padding=True, return_tensors='pt')
        scores.append(mdl(**b).logits.softmax(-1)[:, 1].numpy())
S = np.concatenate(scores)

single = solo.values
true_neg = np.array([1 if s[0] == 'negative' else 0
                     for s in d['sentiments'].str.split('|')[single]])
print('agreement with the true sentiment :',
      round(100 * ((S[single] < 0.5).astype(int) == true_neg).mean(), 1), 'percent')
print('correlation with the star rating  :', round(np.corrcoef(S, d['stars'])[0, 1], 3))
print('mean score on negative reviews    :', round(S[single][true_neg == 1].mean(), 3))
print('mean score on positive reviews    :', round(S[single][true_neg == 0].mean(), 3))

agreement with the true sentiment : 71.0 percent
correlation with the star rating  : 0.366
mean score on negative reviews    : 0.252
mean score on positive reviews    : 0.661

A pretrained sentiment classifier agrees with the truth 71.0% of the time on the reviews that raise a single theme, which are the easy ones. On a review that praises the delivery and condemns the sizing it returns one score for both. Sentiment has to be attached to a theme, not to a review, and that means the classifier has to know the theme first.

7dFinal evaluation

months = sorted(d['month'].unique())
vol = {t: int(Y[:, j].sum()) for j, t in enumerate(THEMES)}
negv = {t: int(NEGM[:, j].sum()) for j, t in enumerate(THEMES)}
mv = pd.DataFrame({t: [int(Y[(d['month'] == m).values, j].sum()) for m in months]
                   for j, t in enumerate(THEMES)}, index=months)
growth = (mv.iloc[4:].mean() / mv.iloc[:4].mean()).sort_values(ascending=False)

rank = pd.DataFrame({
    'by mentions': sorted(vol, key=lambda t: -vol[t]),
    'by negative mentions': sorted(negv, key=lambda t: -negv[t]),
    'by fastest growing': list(growth.index)})
print(rank.head(5).to_string())
print()
print('damage and packaging by month:', mv['delivery_damage'].tolist())

With themes assigned, the report can be ranked three ways. They do not agree.

Three different answers to which theme to fix first. Delivery speed tops the volume list and is the least negative theme in the file. Sizing and fit tops the anger list. And Damage and packaging tops the trend list, on 2.88 times its earlier volume.

Mentions run at about 130 a month for four months and then jump to 369 and 381. Across the whole six months it is only 10.6% of reviews, which is why a quarterly total cannot see it and a sample of fifty will not contain enough of it to notice. It is also the theme LDA missed entirely.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not start with topic modelling. It recovered 4 of 10 themes here and missed the one that mattered.
• Do not tune k until the clusters look right. Purity rises with k forever, and even at 20 clusters three real themes were still missing.
• Do not rank by mentions. It puts the least negative theme in the file at the top.
• Do not use the star rating as the sentiment. It correlates with the text at 0.366 and cannot be split by theme.
• Do not report a quarter at a time. The finding in this file is only visible month by month.
• Do not assume labelling is the expensive option. Here it was 2.5 hours and it beat everything else outright.

Reproducibility

What to take from this

• Unsupervised methods find the themes in the text, not the themes in your plan. Those are different lists and only one of them has owners.
• A few hundred labels is an afternoon, not a project. It beat both unsupervised methods here by a wide margin.
• Volume is the least useful ranking. The loudest theme was the calmest one.
• Sentiment belongs to a theme, not to a review. Half the reviews here carry two themes with different sentiment.
• Trend beats level. A theme that tripled is a change with a cause; a theme that is always large is a fact of the category.
• A sample of fifty cannot see a two per cent theme. That is arithmetic, not diligence.

The customer team asked for the themes in the reviews. The themes were never the hard part: an afternoon of labelling gets 91.7% micro F1 and the topic model was never going to produce the list they already work from. The hard part was that three reasonable ways of ranking the same themes give three different answers, and only one of them points at something that broke in 2026 and can be unbroken.