Feature Engineering and Preprocessing: 10 Code-Along Examples

Learn to get data ready for a model by running it. Ten copy-and-run examples covering cleaning, stratified splitting, scaling without leakage, encoding, engineered features, TF-IDF, feature selection, PCA, and a single pipeline.

The feature engineering article covers everything between a raw table and a model that can be trained on it: cleaning, splitting, scaling, encoding, deriving new columns, turning text into numbers, cutting features down, and assembling the lot into something reproducible. The order matters more than any individual technique, because several of these steps leak information from the test set into the training set if you do them in the wrong sequence, and a leak does not raise an error, it raises your score. Every example here runs against the same deliberately messy dataset and the examples are cumulative, so what you finish one with is what the next one starts from. The idea that ties it together arrives in the last example.

Paste this once to build the working dataset:

import numpy as np
import pandas as pd
rng = np.random.default_rng(19)
n = 600
city = rng.choice(["Manchester", "Leeds", "Bristol", "Glasgow"], n, p=[.35, .3, .2, .15])
plan = rng.choice(["basic", "standard", "premium"], n, p=[.5, .35, .15])
tenure = rng.integers(1, 60, n)
support = rng.poisson(1.4, n)
monthly = np.round([rng.normal({"basic": 22, "standard": 41, "premium": 78}[p], 6) for p in plan], 2)
notes = rng.choice([
    "billing issue not resolved", "great service quick response",
    "cancelled after price rise", "happy with the upgrade",
    "slow support long wait", "no complaints at all",
], n)
logit = (-0.4 + 0.55 * support - 0.045 * tenure + 0.012 * monthly
         + (plan == "basic") * 0.7 + rng.normal(0, 0.6, n))
churn = (1 / (1 + np.exp(-logit)) > 0.5).astype(int)
customers = pd.DataFrame({
    "customer_id": range(1, n + 1),
    "city": city, "plan": plan, "tenure_months": tenure,
    "support_tickets": support, "monthly_charge": monthly,
    "total_charge": np.round(monthly * tenure * rng.uniform(0.9, 1.1, n), 2),
    "signup_date": pd.to_datetime("2021-01-01") + pd.to_timedelta(rng.integers(0, 1500, n), unit="D"),
    "note": notes,
    "churned": churn,
})
# the problems a real extract arrives with
customers.loc[rng.choice(n, 25, replace=False), "monthly_charge"] = np.nan
customers.loc[rng.choice(n, 9, replace=False), "city"] = np.nan
customers = pd.concat([customers, customers.iloc[:12]], ignore_index=True)
customers["total_charge"] = customers["total_charge"].astype(str) + " GBP"

1. Cleaning first

Before anything else, three checks: duplicates, missing values and types that are not what they should be.

print("shape:", customers.shape)
print("duplicates:", customers.duplicated(subset="customer_id").sum())
customers = customers.drop_duplicates(subset="customer_id").reset_index(drop=True)
print(customers.isna().sum()[customers.isna().sum() > 0])
print("total_charge dtype before:", customers["total_charge"].dtype)
customers["total_charge"] = pd.to_numeric(
    customers["total_charge"].str.replace(" GBP", "", regex=False), errors="coerce")
print("total_charge dtype after :", customers["total_charge"].dtype)
print("churn rate:", round(customers["churned"].mean(), 3))

shape: (612, 10)
duplicates: 12
after dedupe: (600, 10)
city               9
monthly_charge    25
dtype: int64
total_charge dtype before: object
total_charge dtype after : float64
churn rate: 0.453

Twelve duplicate customers, two columns with gaps, and a numeric column that arrived as text because someone appended a currency. All three are ordinary and all three would break something downstream: duplicates give the model the same customer twice and inflate any validation score, missing values stop most estimators outright, and a string column silently gets one-hot encoded into hundreds of useless columns if you do not notice. Deduplicate on the business key rather than on whole rows, since two genuinely different customers can share every attribute. And note the churn rate, 45 percent, which is the number the next example has to preserve.

2. Split first, and stratify

The train/test split comes before every transformation, and for a classification target it should preserve the class balance.

from sklearn.model_selection import train_test_split
X = customers.drop(columns=["churned", "customer_id"])
y = customers["churned"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=7, stratify=y)
Xu_tr, Xu_te, yu_tr, yu_te = train_test_split(X, y, test_size=0.25, random_state=7)

rows      train 450  test 150
stratified churn rate  train 0.453  test 0.453
unstratified           train 0.444  test 0.480
full dataset           0.453

stratify=y reproduced the 45.3 percent churn rate exactly in both halves, while the unstratified split drifted to 44.4 and 48.0. On a balanced target that difference is cosmetic; on a rare one it is the difference between a usable test set and a meaningless one, because a two percent positive class split at random can easily give you a test set with almost no positives to score against. Dropping customer_id matters too, since an identifier correlated with row order can leak the target in ways that look like brilliant model performance. From here on, every fit happens on X_tr only.

3. Scaling, and what leakage actually looks like

Scalers learn parameters from data, which is why fitting one on everything is a leak rather than a shortcut.

from sklearn.preprocessing import StandardScaler
num_cols = ["tenure_months", "support_tickets", "monthly_charge", "total_charge"]
tr_num = X_tr[num_cols].fillna(X_tr[num_cols].median())
te_num = X_te[num_cols].fillna(X_tr[num_cols].median())   # train's median, on purpose
scaler = StandardScaler().fit(tr_num)          # fit on train
tr_scaled = scaler.transform(tr_num)
te_scaled = scaler.transform(te_num)           # transform test with train's parameters
leaky = StandardScaler().fit(pd.concat([tr_num, te_num]))
print("leaky scaler mean on tenure:", round(leaky.mean_[0], 3),
      "vs honest:", round(scaler.mean_[0], 3))

after scaling (train):  mean [0. 0. -0. 0.]   std [1. 1. 1. 1.]
after scaling (test) :  mean [0.091 0.072 0.045 0.004]  std [1.027 0.979 0.998 1.006]
leaky scaler mean on tenure: 29.107 vs honest: 28.722

The training set comes out with mean zero and standard deviation one by construction, and the test set comes out nearthose values without matching them, which is the correct result: the test set is being measured with the training set’s ruler and any difference is real information about how the two differ. The leaky scaler’s mean differs only in the second decimal place here, which is exactly why this mistake survives review, and on a small test set or a shifting distribution the same mistake is worth several points of imaginary accuracy. The same rule applies to the imputation above, where the test set is filled with the training median.

4. Encoding categories

A model needs numbers, and the encoder has to survive a category it has never seen.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
enc.fit(X_tr[["city"]].fillna("unknown"))
print("categories learned:", enc.categories_[0].tolist())
out = enc.transform(pd.DataFrame({"city": ["Leeds", "Cardiff", "unknown"]}))

categories learned: ['Bristol', 'Glasgow', 'Leeds', 'Manchester', 'unknown']
         city_Bristol  city_Glasgow  city_Leeds  city_Manchester  city_unknown
Leeds               0             0           1                0             0
Cardiff             0             0           0                0             0
unknown             0             0           0                0             1

Cardiff was not in the training data, and handle_unknown="ignore" turned it into a row of zeros instead of raising an error in production six months from now. That argument is the entire reason to prefer scikit-learn’s encoder over pd.get_dummies, which is more convenient in a notebook and has no memory: run it on the test set separately and you get a different number of columns in a different order, which is a class of bug that produces silently wrong predictions rather than a crash. Note also that missing values were turned into their own unknown category rather than dropped, which is often the right call because “we do not know their city” can itself be predictive.

5. Ordinal against nominal

Some categories have an order and encoding them as unordered throws that away.

from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(categories=[["basic", "standard", "premium"]]).fit(X_tr[["plan"]])
demo = pd.DataFrame({"plan": ["basic", "standard", "premium"]})
demo["ordinal"] = oe.transform(demo[["plan"]]).astype(int)

    plan  ordinal
   basic        0
standard        1
 premium        2

one-hot would give three unordered columns instead:
 plan_basic  plan_premium  plan_standard
          1             0              0
          0             0              1
          0             1              0

Passing categories explicitly is the important part, because the default sorts alphabetically and would have encoded basic as 0, premium as 1 and standard as 2, which asserts that standard is further from basic than premium is. The decision is about the variable, not the algorithm: use ordinal encoding when the order is real and the spacing is roughly meaningful, such as size bands or satisfaction ratings, and one-hot when the categories are simply different, such as city. Getting this wrong in either direction costs you, since one-hot on an ordered variable discards information a tree could have used, and ordinal on an unordered one invents a ranking a linear model will believe.

6. Engineering features

The columns that predict best are often ones you compute rather than ones you were given.

feat = X_tr.copy()
feat["charge_per_month_of_tenure"] = (feat["total_charge"] / feat["tenure_months"]).round(2)
feat["tickets_per_year"] = (feat["support_tickets"] / (feat["tenure_months"] / 12)).round(2)
feat["signup_year"] = feat["signup_date"].dt.year
feat["signup_quarter"] = feat["signup_date"].dt.quarter
feat["tenure_band"] = pd.cut(feat["tenure_months"], bins=[0, 12, 24, 48, 60],
                             labels=["<1y", "1-2y", "2-4y", "4y+"])

 tenure_months  total_charge  charge_per_month_of_tenure  tickets_per_year  signup_year tenure_band
            42       1769.21                       42.12              0.86         2021        2-4y
             2         56.72                       28.36              6.00         2021         <1y
            16       1338.41                       83.65              0.00         2024        1-2y

tenure_band
<1y     105
1-2y      82
2-4y     187
4y+       76

Three kinds of derived feature, each fixing a different blind spot. Ratios normalise for size, so tickets_per_year says something support_tickets cannot, because six tickets from a two-month customer and six from a five-year customer are opposite signals and the raw count treats them as identical. Date parts turn a timestamp, which a model cannot use, into seasonality and cohort features it can. And binning turns a continuous variable into groups, which is worth doing when the relationship is not monotonic or when the bands match how the business already thinks. Two cautions: a ratio with a small denominator explodes, and any feature computed from the target is leakage.

7. Text as numbers with TF-IDF

Free text becomes a numeric matrix by counting words and down-weighting the common ones.

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=12, min_df=5, stop_words="english")
tfidf_tr = tfidf.fit_transform(X_tr["note"])
print("vocabulary:", tfidf.get_feature_names_out().tolist())
print("matrix shape:", tfidf_tr.shape)

vocabulary: ['billing', 'cancelled', 'happy', 'issue', 'long', 'price', 'resolved',
             'rise', 'slow', 'support', 'upgrade', 'wait']
matrix shape: (450, 12) | sparsity: 0.826
a note: cancelled after price rise
cancelled    0.577
price        0.577
rise         0.577

Each note became a row of twelve numbers, one per vocabulary word, mostly zeros. TF-IDF is two ideas multiplied: term frequency, how often the word appears in this document, and inverse document frequency, which shrinks words that appear everywhere, so a word in every note carries almost no weight while a word in a handful carries a lot. The three parameters are the ones you will always set. max_features caps the vocabulary and keeps the matrix manageable, min_dfdrops words appearing in fewer than n documents, which removes typos and one-offs, and stop_words removes the grammatical filler. As always, fit on the training set only, or the vocabulary itself is leaked.

8. Cutting features down

More features are not better, and three cheap methods disagree about which ones to keep.

from sklearn.feature_selection import VarianceThreshold, SelectKBest, f_classif
vt = VarianceThreshold(threshold=0.05).fit(num_tr)
sel = SelectKBest(f_classif, k=3).fit(num_tr, y_tr)
scores = pd.Series(sel.scores_, index=num_tr.columns).sort_values(ascending=False)
corr = num_tr.corr().abs()

ANOVA F scores:
tenure_months                 177.56
total_charge                   84.11
support_tickets                83.73
tickets_per_year               37.25
monthly_charge                  1.01
charge_per_month_of_tenure      0.85
kept: ['tenure_months', 'support_tickets', 'total_charge']
most correlated pairs:
  monthly_charge / charge_per_month_of_tenure: 0.883
  tenure_months / total_charge: 0.674

Three different questions. A variance threshold removes columns that barely change, which is a data-quality filter rather than a modelling one and catches constants and near-constants. SelectKBest with f_classif ranks each feature by how strongly it separates the classes on its own, which is fast and univariate, so it cannot see a feature that only matters in combination with another. And the correlation check finds redundancy rather than uselessness, with monthly_charge and my engineered charge_per_month_of_tenure at 0.88, which means keeping both mostly adds noise and instability to a linear model’s coefficients. Note that tenure_months scores highest by a wide margin, which matches how the data was generated.

9. PCA

When features are correlated, principal components repack the same information into fewer, uncorrelated columns.

from sklearn.decomposition import PCA
pca_input = StandardScaler().fit_transform(num_tr)
pca = PCA().fit(pca_input)
print("explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))
print("cumulative              :", np.round(np.cumsum(pca.explained_variance_ratio_), 3))

explained variance ratio: [0.433 0.247 0.197 0.09  0.022 0.011]
cumulative              : [0.433 0.68  0.877 0.967 0.989 1.   ]
components needed for 95%: 4 of 6
shape after PCA: (450, 4)

Six correlated columns compressed into four that carry 96.7 percent of the variance. The scaling on the line above is not optional, because PCA maximises variance and an unscaled total_charge measured in thousands would dominate support_tickets measured in single digits regardless of which is informative. Read the cumulative row to choose the number of components rather than picking one in advance. The cost is interpretability, since a component is a weighted blend of your original columns and nobody can explain “component two” to a stakeholder, which is why PCA belongs in pipelines where prediction is the goal and not in ones where the coefficients are the deliverable.

10. One pipeline, and the model

Everything above belongs in a single object that applies the same transformations, in the same order, fitted on the same rows.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score
pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale",  StandardScaler())]),
     ["tenure_months", "support_tickets", "monthly_charge", "total_charge"]),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]),
     ["city", "plan"]),
    ("txt", TfidfVectorizer(max_features=12, stop_words="english"), "note"),
])
model = Pipeline([("prep", pre), ("clf", LogisticRegression(max_iter=1000))])
model.fit(X_tr, y_tr)
print("features after preprocessing:", model.named_steps["prep"].transform(X_tr).shape[1])
print("train AUC:", round(roc_auc_score(y_tr, model.predict_proba(X_tr)[:, 1]), 3))
print("test  AUC:", round(roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]), 3))

features after preprocessing: 23
train AUC: 0.924
test  AUC: 0.934
['num__tenure_months', 'num__support_tickets', 'num__monthly_charge',
 'num__total_charge', 'cat__city_Bristol', 'cat__city_Glasgow', ...]

Nine raw columns became 23 features and one fit call did all of it. The ColumnTransformer is what makes this possible, routing each group of columns to its own sub-pipeline, and the outer Pipeline then hands the result to the model. The reason this is worth the extra syntax is not tidiness: a pipeline makes leakage structurally difficult, because fit only ever sees the training rows and predict re-applies exactly the fitted transformations, so you cannot accidentally scale the test set with its own parameters. It is also the only sane way to cross-validate, since the whole pipeline is refitted inside every fold.

The model that ties the ten together is that every preprocessing step is a function with parameters, and those parameters are learned from data, which is why the order is not a matter of taste. A scaler learns a mean and a standard deviation. An imputer learns a median. An encoder learns a category list. A vectoriser learns a vocabulary. A PCA learns a rotation. Each of those is a small model fitted to whatever you show it, and the entire discipline of preprocessing reduces to one rule: fit on the training data, transform everything. That single sentence explains why the split comes first in Example 2, why Example 3’s leaky scaler is wrong, why the test set is imputed with the training median, why the encoder must handle unknown categories, and why Example 10’s pipeline is the safest way to express all of it. The steps that are not fitted, dropping duplicates, parsing a string to a number, computing a ratio from two columns in the same row, are the ones you can safely do before the split.

Work through these and you have the article in practice: duplicates, gaps and a numeric column stored as text; a stratified split that holds the class balance to three decimal places; a scaler fitted on train and the leaky version that differs just enough to hide; one-hot encoding with unknown categories handled; ordinal encoding with an explicitly stated order; ratios, date parts and bands; TF-IDF with the three parameters worth setting; variance, univariate scores and correlation as three different reasons to drop a column; PCA on standardised inputs; and a ColumnTransformer pipeline that does all of it in one fit. The habit that follows is a single question asked before every transformation: does this step learn anything from the data, and if it does, has it only been allowed to learn from the training set.

Thanks for reading, Andrei.