Feature Engineering in Python

Raw data almost never arrives in a form a machine learning model can use. Most algorithms demand numeric input arranged as a matrix, and many assume that input is roughly normally distributed, yet real datasets are full of text categories, skewed distributions, missing values, outliers, and messy strings. Feature engineering is the work of bridging that gap, turning raw columns into clean, numeric, well-behaved features. It is also where a great deal of a model’s eventual performance is quietly won or lost, because a mediocre algorithm on well-engineered features routinely beats a sophisticated one on raw data. This article walks the whole pipeline: exploring the data, encoding categories, transforming numbers, handling gaps and outliers, cleaning text, and the train-test discipline that holds it all together.

Exploring the Data First

Before transforming anything, you look. A few lines tell you what you are dealing with.

import pandas as pd
survey = pd.read_csv('survey.csv')
print(survey.head())
print(survey.dtypes)
print(survey.shape)
print(survey.info())

The most useful of these is info(), because it combines the data type of each column with its non-null count in a single view. If the DataFrame has a thousand rows but a column reports only 850 non-null values, you have spotted 150 missing entries with no extra work. The dtypes output is the other early warning system, flagging columns that pandas read as objectwhen you expected numbers, which usually means stray characters are hiding inside.

A practical next step is splitting the columns by type, because numeric and text columns need entirely different treatment.

only_numeric = survey.select_dtypes(include=['int', 'float'])
only_text = survey.select_dtypes(include=['object'])

Think of the dataset as a drawer holding both measuring tools and labels. The numeric selection pulls out the measuring tools you can do arithmetic on immediately, and the object selection pulls out the labels that must be converted to numbers before any model can touch them. The kinds of columns you will meet fall into familiar buckets: continuous values like salary and age, categorical values like country and job title, ordinal values like a one-to-five rating, booleans, and dates. Each calls for a different engineering approach.

The math behind feature engineering: https://datalad.co.uk/the-mathematics-behind-feature-engineering/

Encoding Categorical Variables

A model cannot read the string “USA”; it reads numbers. One-hot encoding solves this by creating a separate binary column for every unique category, where each row carries a one in its own category’s column and zeros everywhere else.

one_hot = pd.get_dummies(survey, columns=['Country'], prefix='OH')
print(one_hot.columns)

For N categories this produces N new columns. A closely related approach, dummy encoding, drops the first category to leave N minus one columns.

dummies = pd.get_dummies(survey, columns=['Country'], drop_first=True, prefix='DM')

The reason for dropping one is subtle but important. If a row has zero in every country column, you already know it must belong to the dropped country, so that final column is pure redundancy. In a linear model this redundancy causes perfect multicollinearity, a mathematical problem because one column is always predictable from the others, and drop_first=Trueremoves it cleanly. The rule of thumb is simple: use full one-hot encoding for tree-based models, which do not care, and dummy encoding for linear models, which do.

A separate problem arises when categories are rare. If only two people out of ten thousand listed a particular country, giving it a dedicated column produces a feature that is almost entirely zeros, adding noise without signal. The fix is to collapse the rare values into a single catchall.

countries = survey['Country']
counts = countries.value_counts()
rare = counts[counts < 10].index
countries[countries.isin(rare)] = 'Other'

You count each country, identify those appearing fewer than ten times, and relabel all of them as “Other,” turning dozens of tiny categories into one bucket the model can actually learn from. This is part of the same category hygiene covered in the categorical data article.

Transforming Numeric Data

Sometimes the exact magnitude of a number matters less than a simpler distinction. Binarizing collapses a numeric column into a zero-or-one flag.

survey['Paid'] = 0
survey.loc[survey['Salary'] > 0, 'Paid'] = 1

You pre-fill the column with the default of zero, then use .loc with a condition to flip just the qualifying rows to one. Whether someone earns a salary at all may be more predictive than whether they earn 47,000 or 52,000.

Binning goes a step further, grouping a continuous column into discrete buckets, which is like turning a precise thermometer reading into “cold, warm, or hot.”

import numpy as np
survey['salary_band'] = pd.cut(survey['Salary'], bins=5)
bins = [-np.inf, 10000, 50000, 100000, 150000, np.inf]
labels = ['Very low', 'Low', 'Medium', 'High', 'Very high']
survey['salary_label'] = pd.cut(survey['Salary'], bins=bins, labels=labels)

Passing an integer to bins lets pandas calculate equal-width ranges automatically, while a custom list lets you draw the boundaries where domain knowledge says they belong. The -np.inf and np.inf at the ends guarantee that no value, however extreme, falls outside every bucket.

When the actual scale matters, two scalers dominate. MinMax normalization rescales everything linearly into a zero-to-one range.

from sklearn.preprocessing import MinMaxScaler
mm_scaler = MinMaxScaler()
mm_scaler.fit(numeric_df[['Age']])
numeric_df['Age_MM'] = mm_scaler.transform(numeric_df[['Age']])

The formula is just the value minus the minimum, divided by the range, so the youngest person becomes zero, the oldest becomes one, and everyone else lands proportionally between. Its weakness is sensitivity to outliers, because a single extreme value drags the minimum or maximum and squashes everyone else. Standardization, the z-score, sidesteps that.

from sklearn.preprocessing import StandardScaler
ss_scaler = StandardScaler()
ss_scaler.fit(numeric_df[['Age']])
numeric_df['Age_SS'] = ss_scaler.transform(numeric_df[['Age']])

This expresses each value as how many standard deviations it sits above or below the mean, centring the data at zero. It is far more resilient to outliers, because one extreme value barely moves a mean and standard deviation computed across thousands of records.

For data with a long right tail, like salaries where most people cluster low and a few executives earn millions, a power transform reshapes the distribution toward a bell curve.

from sklearn.preprocessing import PowerTransformer
pow_trans = PowerTransformer()
pow_trans.fit(numeric_df[['Salary']])
numeric_df['Salary_LG'] = pow_trans.transform(numeric_df[['Salary']])

PowerTransformer finds the mathematical power that makes the distribution as symmetric as possible and applies it, defaulting to the Yeo-Johnson method, which handles zeros and negatives that a raw logarithm cannot. Plotting a histogram before and after makes the effect obvious: the heavy skew flattens into something far closer to normal. Before reaching for any of these, it pays to actually see the data with histograms, boxplots, and a seaborn pairplot, so you choose the transform that fits the shape in front of you rather than guessing.

Handling Missing Values

Spotting gaps starts with the same info() call, supplemented by isnull() to see exactly which cells are empty as a grid of true and false. Once you know where the holes are, you choose between removing and filling.

Listwise deletion removes rows or columns outright.

no_missing_rows = survey.dropna(how='any')
no_missing_cols = survey.dropna(how='any', axis=1)
no_blank_gender = survey.dropna(subset=['Gender'], axis=0)

The default how='any' is the strictest option, discarding any row with a single blank anywhere; axis=1 turns the same logic on columns; and subset is the surgical version, dropping rows only where one specific column is missing. Always check the shape afterward to see how much data the deletion cost you.

Filling is often better than dropping, and the right fill depends on the data. When a missing categorical value is itself meaningful, such as a survey question someone declined to answer, preserve that fact rather than erasing it.

survey['Gender'].fillna(value='Not Given', inplace=True)

“Not Given” is genuinely different from any real answer, and keeping it as its own category retains all the other information about that person. For a continuous column missing values at random, filling with the mean is the safe neutral choice, because adding the average leaves the column’s average unchanged.

survey['RecommendScore'].fillna(survey['RecommendScore'].mean(), inplace=True)
survey['RecommendScore'] = round(survey['RecommendScore'])

The rounding matters when the column holds whole-number ratings, since the mean of integers is usually a decimal that you want to snap back to a whole number. The broader logic of cleaning gaps and inconsistencies is covered more fully in the data cleaning article.

Removing Outliers

There are two philosophies for trimming extreme values. The percentage-based approach always removes a fixed slice.

threshold = numeric_df['Salary'].quantile(0.95)
trimmed = numeric_df[numeric_df['Salary'] < threshold]

This keeps only rows below the 95th percentile, cutting exactly the top five percent regardless of whether those values are truly anomalous. It is a blunt instrument that will discard genuinely high earners along with any errors. The statistical approach is more discriminating, removing only values that are genuinely far from the rest.

mean = numeric_df['Salary'].mean()
std = numeric_df['Salary'].std()
cut_off = std * 3
trimmed = numeric_df[
    (numeric_df['Salary'] < mean + cut_off) &
    (numeric_df['Salary'] > mean - cut_off)
]

Three standard deviations from the mean covers about 99.7 percent of normally distributed data, so anything beyond that window is genuinely rare. The crucial difference from the percentage method is that if the data is well-behaved, this removes nothing at all, whereas the percentage method always cuts its fixed share whether or not the values deserve it.

The Train-Test Discipline

Here is the rule that prevents one of the most common and damaging mistakes in machine learning: fit any scaler or outlier threshold on the training data only, then apply it to the test data. Never refit on the test set.

from sklearn.preprocessing import StandardScaler
ss_scaler = StandardScaler()
ss_scaler.fit(train_df[['Age']])
test_df['Age_SS'] = ss_scaler.transform(test_df[['Age']])

If you fit the scaler on the full dataset before splitting, the scaler has effectively peeked at the test set, learning its mean and standard deviation and baking that knowledge into the training data. That is data leakage, and it inflates your performance estimate. Fitting on training data and applying those saved statistics to the test set mirrors exactly what happens in production, where future data is scaled using statistics learned from historical training data. The same principle governs outlier removal: derive the boundaries from training statistics, then apply those fixed boundaries to filter the test set, because computing thresholds from the test data would be leakage too.

Cleaning Stray Characters

Numeric columns sometimes arrive as strings polluted with currency symbols and separators. Python cannot turn “50,000” into a number while the comma is there, so it has to go.

survey['SalaryText'] = survey['SalaryText'].str.replace(',', '')

When a conversion fails and you cannot see why, pd.to_numeric with errors='coerce' turns every unconvertible value into NaN instead of crashing, so you can find the culprits.

numeric_vals = pd.to_numeric(survey['SalaryText'], errors='coerce')
print(survey['SalaryText'][numeric_vals.isna()])

Printing the rows that became NaN shows you exactly which characters are blocking the parse. Once you know them all, method chaining cleans everything in one readable expression.

survey['SalaryText'] = (
    survey['SalaryText']
    .str.replace(',', '')
    .str.replace('$', '')
    .str.replace('£', '')
    .astype('float')
)

The value enters dirty at the top of the chain and exits clean at the bottom, with each .str.replace handing its result to the next step like an assembly line, so that by the time .astype('float') runs, every symbol is gone and the conversion succeeds.

Engineering Features from Text

Text needs its own pipeline, and it starts with cleaning. Real text is full of punctuation, numbers, and casing differences that would fragment the vocabulary, treating “Freedom.” and “freedom” as different tokens.

reviews_df['clean'] = reviews_df['review'].str.replace('[^a-zA-Z]', ' ')
reviews_df['clean'] = reviews_df['clean'].str.lower()

The regular expression [^a-zA-Z] matches anything that is not a letter and replaces it with a space, and lowercasing then unifies the casing so the vocabulary stays consistent. Even before building any vocabulary, you can extract cheap, surprisingly predictive features directly.

reviews_df['char_cnt'] = reviews_df['clean'].str.len()
reviews_df['word_cnt'] = reviews_df['clean'].str.split().str.len()
reviews_df['avg_word_length'] = reviews_df['char_cnt'] / reviews_df['word_cnt']

Character and word counts proxy document length, while average word length quietly distinguishes formal from casual writing, since technical prose tends to use longer words.

The workhorse of text features is the bag-of-words count, where each unique word becomes a column and each cell holds how often that word appears in that document.

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(min_df=0.2, max_df=0.8)
cv_transformed = cv.fit_transform(reviews_df['clean'])
cv_array = cv_transformed.toarray()

Most cells are zero because any one document uses only a fraction of the full vocabulary, which is why the output is a sparse matrix that you convert to a dense array with .toarray(). The two frequency filters do important pruning: min_df=0.2drops words appearing in fewer than twenty percent of documents as too rare to matter, and max_df=0.8 drops words in more than eighty percent as too common to distinguish anything, like “the” and “and.” To attach the result back to your data, wrap it in a DataFrame with the vocabulary as column names and a prefix to avoid collisions, then concatenate.

cv_df = pd.DataFrame(cv_array, columns=cv.get_feature_names_out()).add_prefix('Counts_')
reviews_new = pd.concat([reviews_df, cv_df], axis=1, sort=False)

Raw counts have a flaw, though: they reward common words even when those words carry no meaning. TF-IDF fixes this by weighting each word’s frequency in a document against how often it appears across the whole corpus.

from sklearn.feature_extraction.text import TfidfVectorizer
tv = TfidfVectorizer(max_features=100, stop_words='english')
tv_transformed = tv.fit_transform(reviews_df['clean'])
tv_df = pd.DataFrame(tv_transformed.toarray(), columns=tv.get_feature_names_out()).add_prefix('TFIDF_')

A word that is frequent in one review but rare across all the others scores highly, marking it as distinctive, while words that appear everywhere score near zero no matter how often they occur. That is the difference between counting words and measuring which words actually characterise a document.

Single words lose context, since “not great” is two separate tokens to a unigram model even though together they mean the opposite of “great.” N-grams capture sequences instead.

trigram_vec = CountVectorizer(max_features=100, stop_words='english', ngram_range=(3, 3))
trigrams = trigram_vec.fit_transform(reviews_df['clean'])

With ngram_range=(3, 3) the vocabulary becomes three-word phrases like “great value money” or “would buy again,” and a range of (1, 2) would capture single words and pairs together. The most common phrases in a product-review corpus look nothing like those in a legal or medical one, which makes n-grams powerful discriminating features.

Finally, text vectorizers obey the same train-test discipline as scalers. Fit the vectorizer on training text only with fit_transform, then apply it to test text with transform alone.

tv = TfidfVectorizer(max_features=100, stop_words='english')
tv.fit_transform(train_reviews['clean'])
test_vecs = tv.transform(test_reviews['clean'])

Fitting on all documents would let the IDF weights absorb the test set’s word frequencies, information the model should never see. Words that appear only in the test set are silently ignored, exactly as they would be in production where the model was trained long before deployment.

The Pitfalls That Recur

A handful of mistakes account for most feature-engineering bugs. Fitting a scaler on the entire dataset before splitting leaks the test distribution into training, so always split first. Using full one-hot encoding with a linear model invites multicollinearity, where dummy encoding is the fix. Filling missing continuous values with a mean computed across all data, rather than from the training set alone, is the same leakage in a different costume. Leaving rare categories as individual columns adds noise, so group them into “Other.” Running pd.get_dummies separately on test data can produce mismatched column sets, which a fitted encoder avoids. Log-transforming data with zeros or negatives breaks a raw logarithm, which is why PowerTransformer exists. And fitting a vectorizer on all data lets the test set’s vocabulary leak in.

Conclusion

Feature engineering is the bridge between messy raw data and the clean numeric matrix a model expects. Explore first with info() and select_dtypes, encode categories with one-hot for trees and dummies for linear models while collapsing rare values, and transform numbers by binarizing, binning, scaling with MinMax or standardization, or power-transforming skew away. Handle missing values by dropping when you must and filling thoughtfully when you can, trim outliers with the statistical method when you trust the data, and clean stray characters before converting strings to numbers. For text, clean and lowercase, extract cheap length features, then build count, TF-IDF, and n-gram features with sensible frequency filters. Above all, fit every transform on the training data alone and apply it to the test set, because the discipline that prevents leakage is what separates a model that works in a notebook from one that works in the world.