Feature Engineering in Python: 10 Code-Along Examples

The feature engineering guide frames the job plainly: models expect a clean numeric matrix, and raw data never arrives as one. Feature engineering is the bridge. This workbook crosses it ten times: exploring types, encoding categories, binarizing and binning, repairing missing values, rescuing numbers trapped in strings, scaling, taming skew, removing outliers, and turning free text into features. The rule that outranks everything else arrives in Example 10, and the guide states it without hedging: fit your transformers on the training data only.

1. Explore before you engineer

Every transformation starts with knowing what you have. info, dtypes, and select_dtypes split a mixed DataFrame into what is already numeric and what still needs work.

import pandas as pd
import numpy as np
survey = pd.DataFrame({
    "country": ["UK", "US", "UK", "DE", "US", "FR"],
    "salary": [42000, 55000, np.nan, 48000, 61000, 39000],
    "role": ["analyst", "engineer", "analyst", "manager", "engineer", "analyst"],
    "recommend_score": [8, 9, 7, np.nan, 10, 6],
})
survey.info()                                 # types, non-null counts, memory
print(survey.dtypes)
numeric_cols = survey.select_dtypes(include="number").columns
text_cols = survey.select_dtypes(include="object").columns
print("numeric:", list(numeric_cols))         # ['salary', 'recommend_score']
print("text:   ", list(text_cols))            # ['country', 'role']

info() answers three questions at once: what type is each column, how many values are missing, and how big is the frame. The select_dtypes split then defines the work ahead, since the text columns need encoding (Example 2) and the numeric ones need missing-value repair and scaling (Examples 4 and 6). Exploration is not a preliminary; it is the plan.

2. One-hot versus dummy encoding

Models cannot digest the string “UK”, so categories become binary columns. One-hot creates one column per category; dummy encoding drops the first, because for linear models the full set is redundant and causes multicollinearity.

import pandas as pd
survey = pd.DataFrame({"country": ["UK", "US", "UK", "DE", "US", "FR", "ES"]})
# one-hot: N columns for N categories
onehot = pd.get_dummies(survey, columns=["country"], prefix="OH")
print(onehot.head(3))
# dummy: N-1 columns; the dropped category is implied by all zeros
dummy = pd.get_dummies(survey, columns=["country"], drop_first=True)
print(dummy.head(3))
# collapse rare categories first, so encoding doesn't explode
counts = survey["country"].value_counts()
rare = counts[counts < 2].index                     # FR and ES appear once
survey["country_grouped"] = survey["country"].replace(dict.fromkeys(rare, "Other"))
print(pd.get_dummies(survey["country_grouped"]).columns.tolist())
# ['DE', 'Other', 'UK', 'US']

The choice is model-driven: tree models are happy with one-hot, while linear models prefer drop_first=True, since knowing five of six categories determines the sixth. The rare-category collapse matters just as much: without it, a column with fifty one-off values becomes fifty near-empty features that teach a model nothing.

3. Binarizing and binning

Sometimes the exact number matters less than a threshold or a bucket. Binarizing turns a count into a yes-or-no flag, and pd.cut groups a continuous range into labelled bins.

import pandas as pd
orders = pd.DataFrame({"support_tickets": [0, 3, 0, 1, 7, 0],
                       "age": [23, 41, 35, 58, 30, 67]})
# binarize: did this customer EVER contact support?
orders["contacted_support"] = (orders["support_tickets"] > 0).astype(int)
# bin: group ages into labelled brackets
orders["age_band"] = pd.cut(orders["age"],
                            bins=[0, 30, 50, 100],
                            labels=["under_30", "30_to_50", "over_50"])
print(orders)
print(orders["age_band"].value_counts())

The flag says the distinction between zero and any tickets carries the signal, not the ticket count itself. The bins deliberately throw away precision to gain robustness: a model that learns “over 50” generalises where one that memorised “age 58” overfits. Both moves encode judgement about what matters, which is the essence of feature engineering.

4. Missing values: drop or fill

Missing data has three treatments: drop the rows, fill categorical gaps with an explicit label, and fill numeric gaps with a statistic. Which one is right depends on why the value is missing.

import pandas as pd
import numpy as np
survey = pd.DataFrame({
    "country": ["UK", np.nan, "US", "DE", np.nan],
    "salary": [42000, 55000, np.nan, 48000, 61000],
})
print(survey.isna().sum())                     # count the damage first
# option 1: drop any row with a gap (loses 3 of 5 rows here!)
dropped = survey.dropna(how="any")
print(len(dropped), "rows survive dropping")
# option 2: categorical gaps become an explicit category
survey["country"] = survey["country"].fillna("Not Given")
# option 3: numeric gaps take the column mean, rounded back to sensible units
mean_salary = survey["salary"].mean()
survey["salary"] = survey["salary"].fillna(round(mean_salary))
print(survey)

Dropping is honest but expensive, losing three-fifths of this tiny dataset. The “Not Given” label treats absence as information, which it often is, since people who skip the salary question differ from those who answer it. And the mean-fill keeps the row while barely disturbing the column’s statistics. The guide’s leakage warning already applies here: that mean should come from training data only, a discipline Example 10 makes explicit.

5. Rescuing numbers trapped in strings

Real columns arrive as "£42,000" and "unknown". The rescue is a chain: strip the symbols, then convert with errors="coerce" so unparseable values surface as NaN instead of crashing the pipeline.

import pandas as pd
raw = pd.DataFrame({"salary": ["£42,000", "£55,500", "unknown", "£48,250", "N/A"]})
cleaned = (raw["salary"]
           .str.replace("£", "", regex=False)     # kill the currency symbol
           .str.replace(",", "", regex=False))    # kill the thousands separator
raw["salary_num"] = pd.to_numeric(cleaned, errors="coerce")   # junk -> NaN, not a crash
print(raw)
print(raw["salary_num"].isna().sum(), "values could not be parsed")
# the NaNs are now ordinary missing values: Example 4 takes over

The method chain reads like an assembly line, each .str.replace removing one obstacle, and errors="coerce" is the crucial setting: instead of one bad value killing the conversion, the failures become NaN you can count, inspect, and then treat with the missing-value tools you already have. Messy strings are just missing values wearing a disguise.

6. Scaling: MinMax versus standardization

Features on wildly different scales distort distance-based models. MinMax squeezes everything into 0 to 1; standardization recentres on zero with unit variance and shrugs off outliers far better.

import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler
df = pd.DataFrame({"salary": [39000, 42000, 48000, 55000, 61000, 250000],  # one outlier
                   "score": [6, 7, 8, 8, 9, 10]})
minmax = MinMaxScaler().fit_transform(df)
standard = StandardScaler().fit_transform(df)
out = pd.DataFrame({
    "salary": df["salary"],
    "minmax": minmax[:, 0].round(3),
    "standard": standard[:, 0].round(3),
})
print(out)
# minmax: the outlier claims 1.0 and CRUSHES everyone else below 0.11
# standard: normal salaries spread reasonably; the outlier sits ~2.2 sds out

Look at what the £250,000 outlier does: under MinMax it takes the value 1.0 and compresses every ordinary salary into a sliver near zero, destroying the feature’s resolution. Standardization keeps the ordinary values usefully spread. The rule of thumb follows directly: MinMax when you know the bounds and the data is clean, standardization as the safer default.

7. Power transforms for skewed data

Salaries, prices, and counts pile up at the low end with a long right tail, and many models prefer symmetric inputs. The Yeo-Johnson power transform reshapes skew toward normality.

import numpy as np
import pandas as pd
from sklearn.preprocessing import PowerTransformer
rng = np.random.default_rng(0)
salaries = pd.DataFrame({"salary": rng.lognormal(mean=10.6, sigma=0.5, size=1000)})
print("skew before:", round(salaries["salary"].skew(), 2))    # ~1.6: long right tail
pt = PowerTransformer(method="yeo-johnson")
salaries["salary_pt"] = pt.fit_transform(salaries[["salary"]])
print("skew after: ", round(salaries["salary_pt"].skew(), 2))  # ~0: roughly symmetric
print(salaries.describe().round(2))

The skew statistic tells the story: 1.6 before, effectively zero after. Yeo-Johnson earns its place over a plain log transform because it handles zeros and negatives, which real columns contain and which np.log cannot digest, one of the pitfalls the guide lists. Transform, check the skew again, and the long tail is gone.

8. Outlier removal: percentiles and standard deviations

Two standard defences against extreme values: cut everything beyond a percentile, or keep only values within three standard deviations of the mean. They express different beliefs about the data.

import numpy as np
import pandas as pd
rng = np.random.default_rng(1)
df = pd.DataFrame({"salary": np.append(rng.normal(48000, 8000, 197),
                                       [180000, 220000, 300000])})   # 3 planted outliers
# method 1: trim above the 95th percentile (always removes ~5%)
q95 = df["salary"].quantile(0.95)
trimmed = df[df["salary"] < q95]
# method 2: keep within 3 standard deviations (removes only true extremes)
mean, sd = df["salary"].mean(), df["salary"].std()
within3 = df[(df["salary"] - mean).abs() < 3 * sd]
print(f"original: {len(df)} rows")
print(f"95th percentile cut: {len(trimmed)} rows (always drops ~5%)")
print(f"3-sigma cut: {len(within3)} rows (drops only the genuine extremes)")

The percentile method always removes its fixed share, five percent here, even if every value is legitimate. The three-sigma method adapts: well-behaved data loses almost nothing, and only the planted extremes fall. That is why the guide leans toward the statistical method when the data is roughly normal, with the percentile cut as the blunt instrument for badly behaved columns.

9. Text into features

Free text becomes features in three steps: clean it with regex, measure it with length statistics, then vectorize it. CountVectorizer counts words, and TF-IDF reweights them so common words stop dominating.

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
reviews = pd.DataFrame({"text": [
    "Great keyboard, fast delivery!!",
    "The keyboard broke after 2 weeks. Terrible.",
    "Fast delivery, great price, great product.",
]})
# step 1: clean: lowercase, strip punctuation and digits
reviews["clean"] = (reviews["text"]
                    .str.lower()
                    .str.replace(r"[^a-z\s]", "", regex=True))
# step 2: length features
reviews["char_count"] = reviews["clean"].str.len()
reviews["word_count"] = reviews["clean"].str.split().str.len()
print(reviews[["char_count", "word_count"]])
# step 3: TF-IDF: frequent-in-this-doc, rare-in-the-corpus wins
tfidf = TfidfVectorizer(min_df=1)
matrix = tfidf.fit_transform(reviews["clean"])
features = pd.DataFrame(matrix.toarray().round(2),
                        columns=tfidf.get_feature_names_out())
print(features[["great", "broke", "delivery"]])
# "broke" scores high in review 2 only: rare in the corpus, so it carries signal

The TF-IDF weights show the logic: “great” appears everywhere so its weight deflates, while “broke” appears in one review and scores heavily there, exactly the word a churn model should notice. The min_df and max_df arguments prune words too rare or too common to help, and ngram_range=(2,2) would capture pairs like “broke after” when single words lose the meaning.

10. The rule above all: fit on train only

Every fitted transformer, scaler, imputer, vectorizer, learns statistics from data. Learn them from the full dataset and the test set has leaked into training, inflating every score you report. The discipline is mechanical: fit_transform on train, transform on test.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(2)
df = pd.DataFrame({"salary": rng.normal(48000, 9000, 200)})
train, test = train_test_split(df, test_size=0.25, random_state=42)
# WRONG: the scaler has seen the test set's mean and spread
leaky = StandardScaler().fit(df)                      # fit on EVERYTHING
# RIGHT: learn statistics from train, apply them unchanged to test
scaler = StandardScaler()
train_scaled = scaler.fit_transform(train)            # fit + transform on train
test_scaled = scaler.transform(test)                  # transform ONLY on test
print("train mean used:", round(scaler.mean_[0]))     # learned from train alone
print("test mean after scaling:", round(test_scaled.mean(), 3))
# not exactly 0, and that is CORRECT: test must not influence the statistics

The tell is in the last line: the scaled test set’s mean is close to zero but not exactly zero, because the scaler’s statistics came from training data alone, exactly as they would in production where tomorrow’s data cannot influence today’s model. The same fit-on-train rule governs the mean-fill from Example 4, the outlier thresholds from Example 8, and the vectorizer from Example 9. Never refit on the test set is not a preference; it is what keeps every reported score honest.

Work through these and you have the whole article in practice: exploration, both encodings with rare-category collapsing, binarizing and binning, the three missing-value treatments, string rescue with coercion, both scalers and their outlier behaviour, power transforms for skew, two outlier defences, text features through TF-IDF, and the train-only fitting discipline that binds them all. The mindset the guide leaves you with is the one to keep: features are arguments about what matters, and every transformation is a small hypothesis about where the signal lives.

Hope this helps.