Dimensionality Reduction in Python: 10 Code-Along Examples

The dimensionality reduction guide frames the problem plainly: too many features overfit models, slow training, inject noise, and defeat visualisation. The cure splits into two families that this workbook runs in order. Feature selection keeps a subset of your original columns and preserves their meaning. Feature extraction builds new columns from combinations of the old and trades interpretability for compression. By the end you will have thinned a feature set with variance, correlation, and missing-value filters, ranked features three different ways, run PCA properly inside a pipeline, and reconstructed images from a handful of learned components. The rule that governs all of it arrives in Example 4: fit every selector on the training data only.

1. Remove features that cannot help: zero and low variance

A column holding the same value in every row tells a model nothing, and a near-constant column tells it almost nothing. VarianceThreshold removes them, but only after scaling, because variance depends on units.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import MinMaxScaler
from sklearn.feature_selection import VarianceThreshold
X, _ = load_breast_cancer(return_X_y=True)
# plant a constant column and a near-constant one
X = np.column_stack([X, np.ones(X.shape[0]), np.zeros(X.shape[0]) + 0.001])
print("features before:", X.shape[1])
# normalise first, so variance is comparable across features
X_scaled = MinMaxScaler().fit_transform(X)
selector = VarianceThreshold(threshold=0.01)   # drop features varying almost none
X_reduced = selector.fit_transform(X_scaled)
print("features after: ", X_reduced.shape[1])
print("dropped indices:", np.where(~selector.get_support())[0])

The constant column has exactly zero variance and the near-constant one has almost none, so both fall. The reason for scaling first is subtle but decisive: on raw data a feature measured in thousands has enormous variance and a feature measured in decimals has tiny variance, regardless of how informative either is, so a variance threshold on unscaled data would drop the wrong columns. Normalise, then threshold.

2. See the redundancy: a low-dimensional look with t-SNE

Before removing features numerically, it helps to see the structure. t-SNE compresses many dimensions into two so you can eyeball whether the classes even separate, which tells you how much signal the features carry.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
digits = load_digits()                     # 64 features per image
coords = TSNE(n_components=2, learning_rate="auto",
              init="pca", random_state=42).fit_transform(digits.data)
plt.figure(figsize=(8, 6))
sc = plt.scatter(coords[:, 0], coords[:, 1], c=digits.target, cmap="tab10", s=8)
plt.colorbar(sc, label="digit")
plt.title("t-SNE of the 64-dimensional digits")
plt.savefig("tsne_reduction.png", dpi=150, bbox_inches="tight")
print("saved tsne_reduction.png - ten separated islands means the features carry signal")

Ten clear islands mean the 64 pixel features contain more than enough information to tell the digits apart, so aggressive reduction is safe. Two cautions the guide stresses: t-SNE has fit_transform but no separate transform, so it cannot map new points and is useless in production, and its axes are meaningless. It is a diagnostic you look at, never a step you deploy.

3. Drop features that are mostly missing

A column that is 60 percent empty carries little and imputing it invents most of its values. The missing-value ratio filter removes columns above a chosen missingness.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
X, _ = load_breast_cancer(return_X_y=True)
df = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
# punch holes: two columns made heavily missing
rng = np.random.default_rng(0)
df["f5"] = df["f5"].mask(rng.random(len(df)) < 0.60)     # 60% missing
df["f9"] = df["f9"].mask(rng.random(len(df)) < 0.45)     # 45% missing
missing_ratio = df.isna().mean()
print(missing_ratio[missing_ratio > 0].round(2))
# keep only columns missing less than 50%
keep = missing_ratio[missing_ratio < 0.50].index
df_reduced = df[keep]
print("features before:", df.shape[1], "-> after:", df_reduced.shape[1])

The 60 percent column is dropped and the 45 percent column survives the 50 percent cut, which is the judgement call the threshold encodes. The reasoning is about how much you would be inventing: keeping a mostly-empty column means an imputer supplies the majority of its values, so the feature becomes more fabrication than data. The threshold is a dial on how much fabrication you will tolerate.

4. The rule above all: select on training data only

Every selector learns something from data, a variance, a correlation, a ranking. Learn it from the full dataset and the test set has leaked into your feature choice, inflating every score. The discipline is mechanical: split first, fit the selector on train, apply it to test.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.feature_selection import VarianceThreshold
X, y = load_breast_cancer(return_X_y=True)
# split BEFORE any selection
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
scaler = MinMaxScaler().fit(X_train)             # statistics from train alone
selector = VarianceThreshold(0.01).fit(scaler.transform(X_train))
# apply the SAME fitted transforms to test
X_train_sel = selector.transform(scaler.transform(X_train))
X_test_sel = selector.transform(scaler.transform(X_test))
print("kept", X_train_sel.shape[1], "of", X.shape[1], "features")
print("test set transformed with train's rules, not its own")

The order is the whole point. If you had scaled and thresholded on the full dataset before splitting, the test rows would have influenced which features survive, and the reported accuracy would be optimistic for the same reason a leaky scaler is: the evaluation data helped make the model. Split, then select, and the test set stays honest. Every remaining example assumes this discipline even where the split is omitted for brevity.

5. Remove one of every correlated pair

Two features correlated at 0.98 carry nearly the same information, so one is redundant. A masked correlation matrix finds the pairs, and you drop one from each.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
corr = df.corr().abs()
# keep only the upper triangle, so each pair is counted once
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
# flag any feature correlated above 0.95 with an earlier one
to_drop = [col for col in upper.columns if (upper[col] > 0.95).any()]
print("dropping", len(to_drop), "redundant features:")
print(to_drop)
df_reduced = df.drop(columns=to_drop)
print("features before:", df.shape[1], "-> after:", df_reduced.shape[1])

The upper-triangle mask is what stops double counting: without it, a pair appears twice and the symmetric matrix would flag both members, potentially dropping both. Masking to one triangle means each redundant pair contributes exactly one casualty. The guide’s warning applies at the interpretation stage: high correlation flags redundancy, but which of the pair to keep is a domain decision, and correlation is not causation.

6. Rank features by model coefficients

A fitted linear model assigns each feature a weight, and on scaled data the size of that weight ranks the features by influence. This is the simplest feature-importance method.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)     # scaling makes weights comparable
y = data.target
model = LogisticRegression(max_iter=5000).fit(X, y)
importance = pd.Series(np.abs(model.coef_[0]), index=data.feature_names)
print("top 8 features by coefficient magnitude:")
print(importance.sort_values(ascending=False).head(8).round(3))

Scaling is not optional here. On raw data a large coefficient might only mean the feature is measured in small units, so the magnitude would rank units rather than importance; standardising puts every feature on the same footing so the weight reflects genuine influence. The ranking then reads directly, and keeping the top k coefficients is a fast, interpretable selection.

7. Recursive feature elimination

Ranking once ignores that features interact. RFE removes the weakest feature, refits, ranks again, and repeats, so each decision accounts for the features still in play.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)
y = data.target
rfe = RFE(estimator=LogisticRegression(max_iter=5000),
          n_features_to_select=5)
rfe.fit(X, y)
ranking = pd.Series(rfe.ranking_, index=data.feature_names)
print("selected features (rank 1):")
print(ranking[ranking == 1].index.tolist())
print("\nfirst few eliminated, in order of removal:")
print(ranking.sort_values(ascending=False).head(4))

RFE is more thorough than a single ranking because importance shifts as features leave. A feature that looked weak alongside a near-duplicate can become important once the duplicate is removed, and only an iterative method notices. The cost is compute, since it refits the model once per elimination, which is why RFECV exists to pick the number to keep by cross-validation rather than by hand.

8. Tree-based importance, and Lasso as a selector

Two more selectors from different principles. A random forest ranks features by how much they reduce impurity across all splits; Lasso’s L1 penalty drives weak coefficients exactly to zero, selecting features as a side effect.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LassoCV
data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)
y = data.target
names = data.feature_names
# random forest impurity importance (no scaling needed for trees, but harmless)
rf = RandomForestClassifier(n_estimators=200, random_state=42).fit(X, y)
rf_imp = pd.Series(rf.feature_importances_, index=names)
# Lasso: alpha chosen by cross-validation; zeros are deselected
lasso = LassoCV(cv=5, random_state=42, max_iter=10000).fit(X, y)
kept = np.array(names)[lasso.coef_ != 0]
print("top 5 by random forest importance:")
print(rf_imp.sort_values(ascending=False).head(5).round(3))
print(f"\nLasso kept {len(kept)} of {len(names)} features, dropped the rest to zero")

The two disagree in useful ways. The forest measures predictive contribution through many nonlinear splits and spreads importance across correlated features; Lasso is linear and tends to pick one of a correlated group and zero the others. Neither is definitive, which is exactly why the guide recommends the next step, combining them.

9. Consensus selection: trust what several methods agree on

Any single selector can be fooled by the quirks of its method. Combining the verdicts of several and keeping the features they agree on produces a more robust set than trusting any one.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression, LassoCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFE
data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)
y = data.target
names = data.feature_names
K = 10                                          # each method votes for its top 10
votes = pd.Series(0, index=names)
# voter 1: logistic coefficients
lr = LogisticRegression(max_iter=5000).fit(X, y)
votes[pd.Series(np.abs(lr.coef_[0]), index=names).nlargest(K).index] += 1
# voter 2: random forest importance
rf = RandomForestClassifier(n_estimators=200, random_state=42).fit(X, y)
votes[pd.Series(rf.feature_importances_, index=names).nlargest(K).index] += 1
# voter 3: RFE
rfe = RFE(LogisticRegression(max_iter=5000), n_features_to_select=K).fit(X, y)
votes[names[rfe.support_]] += 1
# voter 4: Lasso survivors
lasso = LassoCV(cv=5, random_state=42, max_iter=10000).fit(X, y)
votes[names[lasso.coef_ != 0]] += 1
print("features chosen by 3 or 4 of the 4 methods:")
print(votes[votes >= 3].sort_values(ascending=False))

The features every method votes for are the ones whose importance is robust to how you measure it, and those are the safest to keep. Consensus trades a little of each method’s edge for reliability, which is usually the right trade when the goal is a stable feature set rather than a leaderboard score. It also surfaces disagreement: a feature one method loves and three ignore is worth a second look rather than automatic inclusion.

10. Feature extraction: PCA in a pipeline, and image compression

Selection keeps original columns; extraction builds new ones. PCA constructs uncorrelated components ordered by variance, and the cumulative explained variance tells you how many to keep. Wrapping it in a pipeline keeps the scaling leak-free.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
digits = load_digits()
X = digits.data                                  # 64 pixel features
# how many components to reach 90% of the variance?
pca_full = PCA().fit(StandardScaler().fit_transform(X))
cum = np.cumsum(pca_full.explained_variance_ratio_)
k90 = np.argmax(cum >= 0.90) + 1
print(f"{k90} components capture 90% of the variance (from 64 features)")
# a leak-free reduction pipeline
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("pca", PCA(n_components=k90)),
])
X_reduced = pipe.fit_transform(X)
print("reduced shape:", X_reduced.shape)
# compression view: reconstruct the images from the components
pca_img = PCA(n_components=k90).fit(X)
X_compressed = pca_img.inverse_transform(pca_img.transform(X))
error = np.mean((X - X_compressed) ** 2)
print(f"reconstructed 8x8 digits from {k90} patterns, mean squared error {error:.2f}")

Two ideas close the workbook. The cumulative variance curve turns “how many components” from a guess into a reading: keep enough to reach the variance you are willing to preserve, here 90 percent, and the elbow in that curve is where extra components stop paying. And the reconstruction shows what compression means concretely, since inverse_transform rebuilds each image from only the retained components, so a low reconstruction error proves those few patterns held most of the picture. The pipeline is what keeps it honest under cross-validation, recomputing the scaler and the components on each fold’s training data alone.

Work through these and you have the whole article in practice: variance and missing-value and correlation filters, the leak-free split-then-select discipline, three importance rankings plus their consensus, Lasso as an automatic selector, and PCA for both reduction and compression inside a pipeline. The unifying decision the guide leaves you with is selection versus extraction: keep original features when interpretability matters and a business needs to know which columns drove the model, and extract components when raw compression matters more than knowing what each new axis means.

Hope this helps