Cluster Analysis in Python

Clustering is what you do when your data has no labels but you suspect it has structure. It groups similar points together without being told what the groups are, which makes it the workhorse of exploratory analysis: customer segments, document topics, image palettes, anomaly detection. This article covers two of the most widely used approaches through scipy: hierarchical clustering, which builds a tree of merges, and k-means, which assigns points to a fixed number of centroids. The workflow that ties them together is always the same: visualise the raw data, normalise the features, pick a method, choose the number of clusters, fit and assign labels, then interpret what the clusters mean.

Always Look First

Before running any algorithm, plot the data. A scatter plot takes one line and often answers the question on its own.

from matplotlib import pyplot as plt
plt.scatter(x, y)
plt.show()

If you can already see distinct blobs, clustering will find them and the results will be meaningful. If the data looks like a uniform cloud, no algorithm will manufacture structure that is not there, and the plot saves you the wasted effort of trying.

Normalising with whiten

Clustering is built on distance, and distance is distorted the moment your features live on different scales. A wage measured in thousands and a goal count measured in single digits are not comparable as raw numbers, so the wage will dominate every distance calculation purely because of its units. scipy’s whiten fixes this by dividing each feature by its standard deviation, giving every column unit variance.

from scipy.cluster.vq import whiten
goals = [4, 3, 2, 3, 1, 1, 2, 0, 1, 4]
scaled = whiten(goals)

The transformation preserves the shape of the data, the relative pattern of highs and lows is unchanged, and only the scale shifts so that features become comparable. It works regardless of magnitude, so tiny decimals like interest-rate changes normalise just as cleanly as large values, because only the relative spread within a column matters. In practice you apply it to DataFrame columns and store the results as new columns before clustering.

players['scaled_wage'] = whiten(players['wage'])
players['scaled_value'] = whiten(players['market_value'])
print(players[['scaled_wage', 'scaled_value']].describe())

The .describe() check is a useful habit, because after whitening the standard deviation of each column should sit at about one. If it does not, something went wrong upstream.

Hierarchical Clustering

Hierarchical clustering builds a tree by repeatedly merging the two closest clusters, and crucially it lets you choose the number of clusters after seeing that tree rather than before. Two functions from scipy.cluster.hierarchy do the work: linkagecomputes the merge order, and fcluster cuts the tree into flat labels.

from scipy.cluster.hierarchy import linkage, fcluster
import seaborn as sns
Z = linkage(festival[['x_scaled', 'y_scaled']], 'ward')
festival['cluster_labels'] = fcluster(Z, 2, criterion='maxclust')
sns.scatterplot(x='x_scaled', y='y_scaled', hue='cluster_labels', data=festival)
plt.show()

Think of it as building a family tree for your data. linkage finds the two closest points or groups, merges them, finds the next closest, and continues all the way up to one big group, recording every merge in a matrix. fcluster then cuts that tree at the level that yields the number of clusters you asked for, here two, assigning every row an integer label. Seaborn’s huemaps those labels to colours automatically.

The method argument decides how the distance between two clusters is measured, and it changes the results substantially.

Ward asks which merge increases within-cluster variance the least, producing compact balanced groups. Single linkage chains points together through their nearest members, which suits thin elongated shapes. Complete linkage joins clusters by their farthest members, favouring equal-sized compact groups. You pick based on the shape you expect.

To see the tree itself, draw a dendrogram.

from scipy.cluster.hierarchy import dendrogram
dendrogram(Z)
plt.show()

Each leaf at the bottom is a data point, each horizontal bar is a merge, and the height of a bar shows how far apart the merged clusters were. A tall vertical drop before a merge means the two groups were well separated, which is exactly where you want to cut the tree. Short lines mean the groups were similar and a cut there means little. One important caveat: hierarchical clustering scales poorly, roughly quadratically in memory, so on a few thousand rows it already slows noticeably and it is unsuitable for very large datasets. That is where k-means takes over.

K-Means Clustering

K-means places k centroids, assigns each point to its nearest centroid, moves each centroid to the mean of its assigned points, and repeats until the positions stabilise. scipy splits this across two functions: kmeans returns the final centroids, and vq assigns each point to the nearest one.

from scipy.cluster.vq import kmeans, vq
centroids, distortion = kmeans(festival[['x_scaled', 'y_scaled']], 2)
festival['cluster_labels'], _ = vq(festival[['x_scaled', 'y_scaled']], centroids)

The two-function design separates training from prediction, mirroring sklearn’s fit-and-predict pattern through a functional API. kmeans iterates until the centroids settle and returns their coordinates along with the distortion, which is the sum of squared distances from every point to its centre. vq then labels each point by which centroid it is closest to. Lower distortion means tighter clusters, but there is a trap: adding more clusters always lowers distortion, right down to zero when every point is its own cluster. That is why you cannot simply minimise distortion to choose k.

Choosing k with the Elbow Method

The elbow method plots distortion against the number of clusters and looks for the point where adding more clusters stops helping.

import pandas as pd
distortions = []
cluster_range = range(1, 7)
for k in cluster_range:
    centroids, distortion = kmeans(festival[['x_scaled', 'y_scaled']], k)
    distortions.append(distortion)
elbow = pd.DataFrame({'num_clusters': cluster_range, 'distortions': distortions})
sns.lineplot(x='num_clusters', y='distortions', data=elbow)
plt.xticks(cluster_range)
plt.show()

Because distortion always falls as k rises, the trick is to find where the curve bends sharply. Before the elbow, each new cluster gives a big improvement; after it, the gains flatten out, because the real structure has already been captured. That bend is your recommended k. The method is also honest about its limits: run the same loop on uniformly distributed data with no real groups and the curve descends smoothly with no bend at all, which is the data telling you there is nothing to find.

Seeds and the Equal-Size Bias

K-means starts from random initial centroids, so different random starts can produce different results, especially on ambiguous data. On well-separated data any start converges to the same answer, but on borderline cases two starts can settle into two different, equally valid clusterings. Setting a seed before running kmeans fixes the starting positions and makes results reproducible, which matters whenever you report or compare them.

from numpy import random
random.seed([1, 2, 1000])
centroids, distortion = kmeans(festival[['x_scaled', 'y_scaled']], 2)

K-means carries a deeper limitation worth knowing. It assumes roughly spherical, equal-sized clusters, because it minimises total squared distance. Run it on the classic shape made of one large circle and two small ones, and instead of finding the three intuitive shapes, it splits the large circle in half, since two balanced groups have lower total distortion than one big group and two small ones. When your clusters are irregular in shape or size, hierarchical clustering with single or average linkage, or a density-based method, will serve you better.

Visualising Clusters

You can colour clusters by hand with matplotlib, mapping each label to a colour through a dictionary and applying it per point, but that grows tedious with many clusters. Seaborn does the whole job in one line.

sns.scatterplot(x='x_scaled', y='y_scaled', hue='cluster_labels', data=festival)
plt.show()

The hue argument finds every unique label, assigns each a distinct colour, draws the scatter, and adds a legend automatically, which is why it is almost always the better choice for cluster visualisation.

Application: Finding the Dominant Colours in an Image

An image is just a three-dimensional array of pixels, each with red, green, and blue values, so k-means can find its dominant palette by treating every pixel as a point in colour space. First you flatten the image into per-channel lists.

import matplotlib.image as img
poster_image = img.imread('poster.jpg')   # shape: (height, width, 3)
r, g, b = [], [], []
for row in poster_image:
    for red, green, blue in row:
        r.append(red)
        g.append(green)
        b.append(blue)

After whitening these channels and running the elbow method, k-means in three-dimensional colour space answers the question “how many dominant colours does this image have?” A movie poster might resolve to three, a busy photograph to five or six. The final step recovers the actual colours from the whitened centroids.

r_std, g_std, b_std = poster_df[['red', 'green', 'blue']].std()
colors = []
for centroid in centroids:
    scaled_r, scaled_g, scaled_b = centroid
    colors.append((scaled_r * r_std / 255, scaled_g * g_std / 255, scaled_b * b_std / 255))
plt.imshow([colors])
plt.show()

The centroids live in whitened space, so multiplying back by each channel’s standard deviation reverses the whitening, and dividing by 255 converts from the 0-to-255 integer range into the 0-to-1 float range matplotlib expects. plt.imshow([colors]) then renders the recovered palette as a row of swatches.

Application: Clustering Documents with TF-IDF

Text has to become numeric before it can be clustered, and TF-IDF is the standard way to do it, weighting each word by how distinctive it is. The same vectoriser covered in the feature engineering guide applies here.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(min_df=0.1, max_df=0.75, max_features=50, tokenizer=remove_noise)
tfidf_matrix = vectorizer.fit_transform(plots)

The filters trim both extremes: min_df=0.1 drops words too rare to carry signal, and max_df=0.75 drops words so common they distinguish nothing. The result is a matrix where each row is a document and each column is a word’s discriminating weight. Clustering it and reading the top words per cluster reveals the themes.

num_clusters = 2
centroids, distortion = kmeans(tfidf_matrix.todense(), num_clusters)
terms = vectorizer.get_feature_names_out()
for i in range(num_clusters):
    center_terms = dict(zip(terms, list(centroids[i])))
    sorted_terms = sorted(center_terms, key=center_terms.get, reverse=True)
    print(sorted_terms[:3])

Each cluster centre is effectively an average document for its group, so the highest-weighted words in that centre are the most characteristic words for the theme. If the top three are “ship,” “ocean,” and “captain,” the cluster is about nautical stories. Note the .todense() call: TF-IDF produces a sparse matrix, but scipy’s kmeans requires dense input.

Interpreting and Validating Clusters

Producing labels is only half the job; the labels are meaningless integers until you interpret them, and a couple of grouped aggregations do that fastest.

print(players.groupby('cluster_labels')['id'].count())
print(players.groupby('cluster_labels')['wage'].mean())

The count tells you whether the clusters are balanced, since one cluster holding ninety percent of the data is a sign of a poor k or heavily skewed data. The mean tells you what each cluster actually represents, since a group with a much higher average wage is probably a premium segment. The richer interpretation comes from clustering on several features at once and reading the average profile of each group.

skill_features = ['scaled_pace', 'scaled_shooting', 'scaled_passing',
                  'scaled_dribbling', 'scaled_defending', 'scaled_physical']
centroids, _ = kmeans(players[skill_features], 2)
players['cluster_labels'], _ = vq(players[skill_features], centroids)
players.groupby('cluster_labels')[skill_features].mean().plot(kind='bar', legend=True)
plt.show()

Clustering across six attributes simultaneously lets the algorithm find natural archetypes, and a bar chart of mean attributes per cluster is the clearest way to read them: one group might show high pace, shooting, and dribbling with low defending, marking the attackers, while another shows the opposite, marking the defenders. Printing a few real member names per cluster is the final sanity check that the interpretation holds.

Choosing Between the Two Methods

The decision usually comes down to size, shape, and what you want to see. On a small dataset where you want to inspect the merge tree or do not know k upfront, hierarchical clustering with a dendrogram is ideal. On a large dataset, k-means is the practical choice because hierarchical clustering is too slow and memory-hungry. For irregular, non-round shapes, hierarchical clustering with single or average linkage, or a density-based method, handles them better than k-means, which assumes spherical equal-sized clusters. When you need reproducibility, k-means with a fixed seed delivers it. And for the two applied cases, document clustering means TF-IDF feeding k-means on a dense matrix, while image colour clustering means extracting RGB, whitening, and running k-means.

Conclusion

Cluster analysis finds structure in unlabelled data, and the discipline around it matters as much as the algorithm. Always plot first, because some clusters are visible before any computation and some data has no structure to find. Always whiten your features, because distance-based methods are meaningless on mismatched scales. Use hierarchical clustering with a dendrogram on small data when you want to choose k by eye, and k-means with the elbow method and a fixed seed on larger data, remembering that it favours round equal-sized clusters. Whichever you choose, the work is not done at the labels: group by cluster and compare counts and means to turn anonymous integers into segments you can actually name and act on.

See you soon.