Unsupervised Machine Learning: Clustering, Dimensionality Reduction, and Topic Modeling

In supervised learning, every training example carries a label. Unsupervised learning removes that requirement: you hand the algorithm raw data with no predefined categories and ask it to find structure on its own. This article covers the main techniques available in scikit-learn and scipy, from grouping records into clusters to extracting topics from text and building similarity-based recommenders.

K-Means Clustering

K-Means partitions data into k groups by minimizing the distance between each point and the center of its assigned cluster. The algorithm places k centroids at random, assigns each point to its nearest centroid, moves each centroid to the mean of its group, and repeats those two steps until the centroids stop moving.

from sklearn.cluster import KMeans
model = KMeans(n_clusters=3)
model.fit(data)                       # find the three centroids
labels = model.predict(data)          # assign each point (returns 0, 1, or 2)
new_labels = model.predict(new_data)  # assign new points using existing centroids
print(model.cluster_centers_)         # (3, n_features) array of centroid coordinates
df['labels'] = labels

.fit() stores the centroid positions. .predict() computes the nearest centroid for each row and returns an integer label. Once fitted, the model can assign labels to data it has never seen before.

Choosing k: the elbow plot

More clusters always produce tighter groupings. Inertia (the sum of squared distances from each point to its centroid) decreases as k increases and reaches zero when every point is its own cluster, which is not useful. The goal is to find where adding more clusters gives diminishing returns.

inertias = []
for k in range(1, 9):
    model = KMeans(n_clusters=k)
    model.fit(data)
    inertias.append(model.inertia_)
plt.plot(range(1, 9), inertias, '-o')
plt.xlabel('Number of clusters (k)')
plt.ylabel('Inertia')
plt.title('Elbow Plot')
plt.show()

Inertia drops steeply before the elbow and flattens after it. The k at the bend is typically the best practical choice.

The math behind it: https://datalad.co.uk/the-mathematics-behind-unsupervised-learning-pca-and-k-means/

Evaluating clusters against known labels

When ground-truth labels exist, a cross-tabulation shows how well the discovered clusters correspond to the true categories.

ct = pd.crosstab(df['labels'], df['true_labels'])
print(ct)

Rows are cluster numbers assigned by K-Means; columns are the real categories. A well-fitted model shows each row dominated by a single column. A uniformly mixed table means the clusters do not correspond to the real categories.

Preprocessing for Clustering

K-Means uses distance to assign points. Features on different scales distort those distances, giving outsized influence to features with large numeric ranges. Scaling before clustering is not optional.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipeline = make_pipeline(StandardScaler(), KMeans(n_clusters=4))
pipeline.fit(data)
labels = pipeline.predict(data)

For data where direction matters more than magnitude, Normalizer is the right choice. Stock price movements are a good example: what you want to cluster is the pattern of ups and downs across trading days, not the raw dollar amounts. Normalizer rescales each row to unit length, so the model focuses on relative direction rather than scale.

from sklearn.preprocessing import Normalizer
pipeline = make_pipeline(Normalizer(), KMeans(n_clusters=10))
pipeline.fit(price_changes)
labels = pipeline.predict(price_changes)

Hierarchical Clustering

Hierarchical clustering builds a tree of merges rather than committing to a fixed k upfront. Every point starts as its own cluster. The two closest clusters merge, then the next closest, repeating until everything is one group. The resulting tree, called a dendrogram, lets you inspect the data’s structure and choose a cut point after the fact.

Hierarchical clustering comes from scipy, not scikit-learn.

from scipy.cluster.hierarchy import linkage, dendrogram, fcluster
mergings = linkage(data, method='complete')
dendrogram(mergings, labels=row_labels, leaf_rotation=90, leaf_font_size=6)
plt.show()
labels = fcluster(mergings, 15, criterion='distance')

linkage returns a matrix encoding which clusters merged and at what distance. dendrogram draws the tree. fcluster cuts the tree horizontally at a chosen distance threshold and returns flat cluster labels.

Reading a dendrogram: the height of a merge point is the distance at which those two clusters were joined. Tall vertical lines indicate well-separated clusters. A horizontal cut at any height extracts a specific number of groups.

t-SNE

t-SNE (t-distributed Stochastic Neighbor Embedding) maps high-dimensional data to two dimensions for visualization. Points that were close together in the original space end up close on the 2D map; structure that was invisible at high dimension becomes visible as clusters on a scatter plot.

One hard constraint: t-SNE has no .transform() method. It cannot process new data after fitting and cannot be used as a preprocessing step for another model. It is strictly a visualization tool.

from sklearn.manifold import TSNE
model = TSNE(learning_rate=200)
tsne_features = model.fit_transform(data)  # returns (n_samples, 2) array
xs = tsne_features[:, 0]
ys = tsne_features[:, 1]
plt.scatter(xs, ys, alpha=0.5)
for x, y, name in zip(xs, ys, entity_names):
    plt.annotate(name, (x, y), fontsize=5, alpha=0.75)
plt.show()

learning_rate typically sits between 50 and 300. Too low and all points clump into a dense ball; too high and they spread out uniformly with no visible structure. If the plot looks wrong, try a different value and replot.

PCA

Decorrelation

PCA finds the directions of maximum variance in the data, called principal components. The first component points along the axis of greatest spread, the second along the next-greatest direction perpendicular to the first, and so on. Projecting onto these axes removes correlations between features.

from sklearn.decomposition import PCA
pca = PCA()
pca.fit(crops)
pca_features = pca.transform(crops)
print(pca.components_)            # each row is one principal component direction
first_pc = pca.components_[0, :] # loadings for the first component

pca.components_ is a matrix where each row is a unit vector pointing in one principal component direction. The values in each row, called loadings, show how much each original feature contributes to that component.

Intrinsic dimension

Not all components carry meaningful information. Plotting explained variance per component shows how many dimensions the data truly inhabits.

pca = PCA()
pca.fit(data)
plt.bar(range(pca.n_components_), pca.explained_variance_)
plt.xlabel('PCA feature')
plt.ylabel('Variance')
plt.show()

If the first two bars are tall and everything after drops near zero, the data is effectively two-dimensional despite having many original features. That number of non-trivial components is the intrinsic dimension.

Dimensionality reduction

Keeping only the top n components discards noise and redundancy while preserving the most informative structure.

scaler = StandardScaler()
pca = PCA(n_components=2)
pipeline = make_pipeline(scaler, pca)
pca_features = pipeline.fit_transform(data)
print(pca.explained_variance_ratio_)        # e.g. [0.72, 0.15]
print(pca.explained_variance_ratio_.sum())  # e.g. 0.87 total retained

Always scale before PCA. An unscaled large-valued feature would dominate the components regardless of how informative it actually is. explained_variance_ratio_ shows the fraction of total variance captured by each component; summing them gives the total information retained after reduction.

Reading component loadings

Each principal component is a linear combination of the original features. Inspecting the loadings reveals what each component actually represents.

pca = PCA(n_components=2)
pca.fit(scaled_data)
print(pca.components_)
# rows = components, columns = original features

If the first component has large loadings on length, width, and height measurements, it is effectively capturing overall size. Naming components this way turns abstract matrix decomposition into something interpretable.

TF-IDF Vectorization

Raw text cannot go directly into a machine learning model. TfidfVectorizer converts a list of documents into a numeric matrix where each row is a document and each column is a word. Words that appear in nearly every document (and therefore carry little distinguishing information) receive low scores; words that appear frequently in a specific document but rarely elsewhere receive high scores.

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
tfidf_matrix = tfidf.fit_transform(raw_docs)  # returns a sparse CSR matrix
words = tfidf.get_feature_names_out()
print(tfidf_matrix[0, :].toarray())           # dense view of first document's scores

The output is a sparse matrix because any individual document uses only a small fraction of the full vocabulary. Most values are zero. .toarray() converts to a dense NumPy array when needed, but for large vocabularies this can consume significant memory.

TruncatedSVD + K-Means for Text Clustering

A TF-IDF matrix might have tens of thousands of columns, one per word in the vocabulary. Running K-Means directly on that is slow and the high dimensionality adds noise. TruncatedSVD compresses the matrix to a compact set of latent directions first, then K-Means clusters on those.

from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(n_components=50)
kmeans = KMeans(n_clusters=6)
pipeline = make_pipeline(svd, kmeans)
pipeline.fit(tfidf_matrix)
labels = pipeline.predict(tfidf_matrix)

TruncatedSVD plays the same role as PCA on sparse data. PCA centers the data by subtracting the mean, which immediately destroys sparsity and turns a sparse matrix dense. TruncatedSVD skips centering and operates on the sparse structure directly, keeping memory usage practical.

NMF: Non-Negative Matrix Factorization

NMF decomposes a non-negative matrix into two non-negative factor matrices. Because all values stay non-negative, the components can be interpreted as additive parts rather than abstract directions. This makes NMF useful for topic modeling on text and for discovering visual building blocks in images.

Topic modeling

from sklearn.decomposition import NMF
model = NMF(n_components=6)
model.fit(tfidf_matrix)
nmf_features = model.transform(tfidf_matrix)  # shape: (n_docs, n_topics)
components_df = pd.DataFrame(model.components_, columns=words)
for i in range(6):
    component = components_df.iloc[i]
    print(f'Topic {i}:')
    print(component.nlargest())
    print()

model.components_ has shape (n_topics, n_words). Each row is a topic defined by its weights across the vocabulary. The highest-weighted words in a row name that topic: if component 0 scores “election”, “vote”, “senate”, and “ballot” highest, the topic is politics.

nmf_features has shape (n_documents, n_topics). Each row describes how strongly that document belongs to each topic.

Image decomposition

NMF also discovers the building blocks of images, the visual parts that combine additively to reconstruct any example.

model = NMF(n_components=7)
model.fit(led_images)            # shape: (n_samples, n_pixels)
components = model.components_   # shape: (7, n_pixels)
for component in components:
    bitmap = component.reshape(13, 8)   # reshape to image dimensions
    plt.imshow(bitmap, cmap='gray', interpolation='nearest')
    plt.show()

For a dataset of LED digit images, NMF learns the seven display segments that combine to form any digit. Each component visualizes as one fragment of the display, and any complete digit can be reconstructed as a weighted sum of those fragments.

NMF vs PCA

Cosine Similarity and Recommendations

After NMF, each item is represented as a mix of topics. Two items are similar if their topic mixes point in the same direction, regardless of the overall magnitude of their scores. Normalizing each row to unit length and computing dot products gives cosine similarity directly.

from sklearn.preprocessing import normalize
norm_features = normalize(nmf_features)   # each row scaled to unit length
df = pd.DataFrame(norm_features, index=doc_titles)
current_doc = df.loc['Champions League Final']
similarities = df.dot(current_doc)         # dot product of unit vectors = cosine similarity
print(similarities.nlargest())

For unit-length vectors, the dot product equals the cosine of the angle between them: 1.0 means identical topic mix, 0.0 means no shared topics. nlargest() returns the most similar items.

End-to-end recommendation pipeline

from sklearn.preprocessing import MaxAbsScaler, Normalizer
pipeline = make_pipeline(
    MaxAbsScaler(),        # scale without destroying sparse zeros
    NMF(n_components=20),  # decompose into 20 latent topics
    Normalizer()           # unit-length rows for cosine similarity
)
norm_features = pipeline.fit_transform(listen_counts)
df = pd.DataFrame(norm_features, index=artist_list)
similarities = df.dot(df.loc['David Bowie'])
print(similarities.nlargest())

MaxAbsScaler divides each feature by its maximum absolute value without subtracting the mean, so sparse zeros remain zero. This preserves the structure of listening-count data where most users have zero plays for most artists.

The three steps together take raw listening data and produce unit-length topic-mix vectors ready for dot-product similarity, outputting a ranked list of artists whose patterns most closely match the reference.

Quick Reference

Which Technique to Use

Goal?
|
+-- Group into clusters
|   +-- Know how many?                --> KMeans
|   +-- Want to explore structure?    --> Hierarchical + dendrogram
|   +-- Text data?                    --> TF-IDF --> TruncatedSVD --> KMeans
|
+-- Visualize in 2D
|   --> t-SNE
|
+-- Reduce feature count
|   +-- Dense data?                   --> StandardScaler --> PCA
|   +-- Sparse data (text)?           --> TruncatedSVD
|
+-- Find topics or visual parts
|   +-- Data is non-negative?         --> NMF
|   +-- Data has negatives?           --> PCA
|
+-- Build a recommender / find similar items
    --> NMF --> Normalize --> cosine similarity (dot product)

Data type for preprocessing?
|
+-- Numeric, dense, different scales  --> StandardScaler
+-- Direction matters, not magnitude  --> Normalizer
+-- Sparse matrix (e.g. TF-IDF)       --> MaxAbsScaler + TruncatedSVD (not PCA)
+-- Raw text                          --> TfidfVectorizer first

See you soon.