The Mathematics Behind Feature Engineering

Feature engineering is often described as the art of the field, the part that resists automation and rewards intuition. That description is half right and half an excuse. The choices are creative, but the transformations themselves are pure mathematics, and every one of them is a specific operation on a random variable with a specific, derivable effect on its distribution. Standardisation is not a ritual; it is an affine map that sets the first two moments. A log transform is not a trick for “making things more normal”; it is a variance-stabilising transformation with a precise domain of validity. One-hot encoding is not free; it spends dimensions, and the geometry of high-dimensional space charges for them.

This article works through that mathematics. It explains why some models collapse without scaling while others do not notice, why the log transform tames a long tail and exactly when it cannot, what a power transform optimises, and why the single most important rule in the whole subject, fit on the training data only, is a statement about estimators rather than a matter of hygiene. The workbook showed these transformations working. This is why they work.

A feature is a random variable

Fix notation before anything else. A feature is a column, and we treat it as a random variable X with realisations $x_1, \ldots, x_n$. Its mean and variance are:

and in practice we work with their sample estimates:

Every transformation below is a function applied to this variable, and the question we ask of each is the same: what does it do to the distribution, and why does a model care?

Standardisation sets the moments

The z-score transformation is:

It is an affine map, a shift followed by a scale, and its effect on the moments is immediate. Using linearity of expectation and the fact that variance scales by the square of a constant:

So standardisation forces mean zero and variance one, by construction, for any input distribution whatsoever. Two facts about it are worth stating because they are often misunderstood.

First, it does not change the shape. Skewness and kurtosis are invariant under affine maps, since they are defined on standardised moments already. A skewed feature is exactly as skewed after standardising, so anyone reaching for StandardScaler to fix a long tail is using the wrong tool, and the power transform below is the right one.

Second, it is not the same as normalising to a distribution. The name misleads: standardisation makes the first two moments match a standard normal, but the variable is not made normal. Only its location and scale move.

Why scaling decides some models and not others

The practical question is when this matters. The answer is a clean dichotomy that falls out of how each model uses the features.

Consider any distance-based method. The squared Euclidean distance between two points is:

Each feature contributes its squared difference, and those contributions are added as equals. But a feature measured in thousands produces differences in the thousands, and a feature measured in units produces differences near one, so the sum is dominated by whichever feature happens to have the largest scale. The distance is effectively computed on one column. KNN, k-means, and any RBF kernel inherit this directly, which is the mathematics behind the workbook’s result where a single StandardScaler moved k-means from mediocre to near-perfect.

Now consider gradient descent on a linear model. The loss surface as a function of the weights has curvature described by the Hessian, which for squared error is proportional to $X^\top X$. When features have wildly different scales, this matrix is ill-conditioned, meaning its condition number, the ratio of largest to smallest eigenvalue, is large:

Gradient descent’s convergence rate depends on $\kappa$: a large condition number produces a loss surface shaped like a long narrow valley, and the gradient zig-zags across it instead of running down it, so training is slow. Standardising the features equalises the diagonal of $X^\top X$, shrinks $\kappa$, and straightens the valley. This is why neural networks and logistic regression train faster on scaled inputs even though, unlike KNN, they would eventually reach the same solution.

And consider regularisation, from the ridge, lasso, and SVM articles. A penalty like $\lambda \sum_j w_j^2$ charges every weight equally, but a feature on a large scale needs only a tiny weight to have an effect, while a feature on a small scale needs a large one. The penalty cannot tell the difference between a large weight that matters and a large weight forced by small units, so it penalises the small-scale feature’s coefficient unfairly. Regularisation is only coherent on standardised features, which is why scikit-learn’s penalised models assume you have scaled.

The models that genuinely do not care are the tree ensembles. A decision tree splits on thresholds of one feature at a time, $x_j \le t$, and any monotonic rescaling of $x_j$ moves the threshold but not which points fall on each side. Trees are invariant to monotonic transforms of individual features, so scaling a random forest changes nothing. The dichotomy is therefore not arbitrary: anything that mixes features through a sum, a distance, a dot product, or a shared penalty needs scaling; anything that treats features one threshold at a time does not.

Min-max scaling and its fragility

The other common rescaling maps to a fixed interval:

It is also affine, but its constants are the sample extremes rather than the mean and standard deviation, and that is its weakness. A single outlier sets $x_{\max}$, and then every other value is compressed into a sliver near zero. The workbook made this visible: with one salary of £250,000 among values near £50,000, min-max scaling pushed all the ordinary salaries below 0.11 and handed the outlier the entire top of the range.

The contrast with standardisation is a contrast in breaking points. Min-max depends on the extremes, which are the least stable statistics of a sample, so a lone outlier dominates. Standardisation depends on the mean and standard deviation, which absorb an outlier’s influence across all n points. The rule follows: min-max when the bounds are known and meaningful and the data is clean, standardisation as the robust default, and a robust scaler using the median and interquartile range when outliers are present but you still want a bounded feel.

Skewness, and what a long tail does

Money, counts, durations, and sizes share a shape: a pile near zero and a long tail to the right. The formal measure is the third standardised moment, skewness:

Symmetric distributions have $\gamma_1 = 0$; a right tail gives $\gamma_1 > 0$. The cube is what makes it a tail detector: a value far above the mean, once standardised and cubed, contributes enormously, so a handful of large observations dominate the statistic. The workbook’s lognormal salaries scored about 1.6, firmly skewed.

Why does a model care about the third moment? Because many models make an implicit symmetry assumption. Linear regression’s errors are assumed homoskedastic and roughly symmetric; a distance is dominated by the tail when one exists; and a mean, the statistic imputation and many summaries rely on, is dragged toward the tail and stops representing a typical value. On a right-skewed feature the mean exceeds the median, so “fill with the mean” fills with an atypically high number.

The log transform, derived properly

The standard fix is the logarithm:

Its tail-taming power is best understood through what it does to multiplicative structure. Skewed variables are typically multiplicative: incomes, populations, and prices vary by ratios, not by additive amounts, so a jump from 1,000 to 10,000 is “the same size” as one from 10,000 to 100,000. The log converts ratios into differences:

so equal ratios become equal distances, and the compressed low end spreads out while the stretched high end pulls in. A lognormal variable, whose logarithm is normal by definition, becomes exactly normal under the transform, which is why the log is the natural first attempt on positive skewed data.

There is a deeper reason from statistics, the delta method. If a variable’s variance grows with its mean, as counts and incomes do, a transform g that stabilises the variance satisfies approximately:

For data where the standard deviation is proportional to the mean, meaning $\operatorname{Var}(X) \propto \mu^2$, the choice $g = \log$ gives $g'(\mu) = 1/\mu$, and the two factors cancel:

The log is not a heuristic for prettier histograms; it is the variance-stabilising transformation for data whose spread scales with its level, which describes most money and count features you will meet.

But the log has a hard domain. It is undefined at zero and negative, and $\log(0) = -\infty$. Real columns contain zeros, so applying np.log to them fails, which the workbook flagged as a pitfall. The common patch is $\log(x + 1)$, written log1p, which handles exact zeros but is an arbitrary shift that distorts the low end, and it still cannot take negatives. This limitation is exactly what the next transform was built to remove.

Power transforms optimise, rather than assume

The Box-Cox transformation generalises the log into a family indexed by a parameter $\lambda$:

The family is continuous, and the log is its limit. Taking $\lambda \to 0$ recovers the logarithm, because $\frac{x^\lambda – 1}{\lambda} \to \log x$, so $\lambda$ interpolates smoothly between no transform ($\lambda = 1$, up to a shift), the square root ($\lambda = 0.5$), the log ($\lambda = 0$), and the reciprocal ($\lambda = -1$).

The crucial change is that $\lambda$ is not chosen by hand; it is estimated from the data by maximum likelihood, selecting the value that makes the transformed data as close to normal as possible:

where $\ell$ is the log-likelihood of the transformed data under a normal model, including the Jacobian term that accounts for the transform’s stretching of the density. So a power transform does not assume the log is right; it searches a whole family and lets the data pick the member that works. When the log is optimal it finds $\lambda \approx 0$ on its own.

Box-Cox still requires strictly positive input. The Yeo-Johnson transform removes even that constraint by defining separate branches for non-negative and negative values:

The definition looks fearsome but the idea is simple: apply a power-transform shape on each side of zero, chosen so the pieces join smoothly, so the whole real line is covered including zeros and negatives. This is precisely why the workbook reached for Yeo-Johnson rather than a raw log: it handles the zeros and negatives that break np.log, and it fits its own exponent. It took the lognormal salaries from skewness 1.6 to essentially 0.

The dimensionality cost of encoding

Categorical features cannot enter a numeric model directly, and one-hot encoding is the standard bridge. A categorical with k levels becomes k binary columns, or k-1 under dummy encoding to avoid the linear dependence that would otherwise make the design matrix singular. That dependence is real: the k indicator columns always sum to the all-ones vector, so one is redundant, which is the mathematical reason drop_first=True exists for linear models.

The cost is dimensional, and high dimensions are stranger than intuition allows. A categorical with 50 cities becomes 50 columns; a few such features and the feature space has hundreds of axes. Two consequences follow directly from the geometry.

First, sparsity. Each one-hot row has a single 1 among k-1 zeros, so the data is mostly empty, and the number of parameters a linear model must estimate grows with the total number of levels while the information per level does not.

Second, the concentration of distances. In high dimension, the distances between random points become nearly equal. If features are roughly independent, the ratio of the spread of pairwise distances to their average behaves like:

As the dimension p grows, this ratio shrinks toward zero, meaning every point is almost the same distance from every other. Nearest-neighbour and clustering methods rely on some points being meaningfully closer than others, and one-hot encoding a high-cardinality feature can push them into the regime where that distinction has evaporated. This is the concrete mechanism behind “the curse of dimensionality”, and it is why the workbook collapses rare categories into an “Other” bucket before encoding: doing so caps k, keeps the dimension down, and stops the distance signal from washing out.

The alternative for very high cardinality is target or frequency encoding, which replaces the category with a single number, its mean target or its frequency, trading k columns for one. That buys dimension back but introduces a leakage risk, which brings us to the rule that governs everything.

The rule above all, as an estimator problem

Every transformation in this article has a hidden dependency that is easy to miss. Standardisation needs $\mu$ and $\sigma$. Min-max needs the extremes. The power transform needs $\hat{\lambda}$. Mean imputation needs the mean. Target encoding needs the per-category target mean. Each of these is a parameter estimated from data, and the entire discipline of avoiding leakage is the discipline of estimating them from the right data.

State it precisely. We want a model whose performance we can estimate honestly on unseen data. The moment a transform’s parameter is computed from the full dataset before the train-test split, the test set has contributed to that parameter, so information from the test set has entered the training pipeline. The estimator of generalisation error is then biased optimistically, for exactly the reason the model validation article derived: the evaluation data has influenced the thing being evaluated.

The correct procedure treats the transform’s parameters as quantities to be learned only on training data:

In scikit-learn this is fit_transform on train and transform on test, never fit on the whole. The consequence people find surprising, that the scaled test set does not have mean exactly zero, is not a bug but the proof of correctness: its mean was computed with the training set’s $\mu$ and $\sigma$, so it lands near zero but not on it, exactly as it would in production where tomorrow’s data cannot be used to standardise itself.

Cross-validation makes the same demand more sharply. If you standardise once before cross-validating, every fold’s held-out data has already leaked into the scaler, and every fold’s score is optimistic. The parameters must be re-estimated inside each fold, on that fold’s training portion alone, which is precisely what a Pipeline guarantees and manual preprocessing silently violates. The pipeline is not tidiness; it is the object that makes the leakage impossible.

Target encoding is where this bites hardest. Replacing a category with its mean target, computed over all rows, means each row’s feature partly encodes its own label, and the model can then “predict” the training labels almost perfectly while learning nothing that generalises. The fix is to estimate the category means out-of-fold, so no row contributes to the mean used for its own encoding, which is leakage avoidance applied to a transform whose parameter is the target itself.

What the mathematics tells you to do

Scale before any distance, dot-product, or penalised model, because those combine features through a sum and the sum has no idea your columns are in different units; skip it for tree ensembles, because a threshold split is invariant to monotonic rescaling. Prefer standardisation to min-max as the default, because the extremes that min-max depends on are the least stable statistics a sample has. Check skewness on money and count features, because the third moment is where the long tail hides and a skewed feature drags every mean-based summary toward it. Reach for a power transform rather than a bare log, because Yeo-Johnson fits its own exponent and survives the zeros and negatives that make np.log throw. Collapse rare categories before one-hot encoding, because dimension is not free and distances concentrate as it grows. And fit every transform on training data alone, because each transform carries an estimated parameter, and estimating it from data you will later evaluate on is the definition of leakage.

The unifying idea is that feature engineering has less art in it than its reputation suggests. A feature is a random variable, a transformation is a function that acts on its distribution in a knowable way, and choosing a transform is choosing what shape your model needs to see. Standardisation sets the moments, the log stabilises multiplicative variance, the power transform searches for the exponent that restores symmetry, and encoding spends dimensions you then have to afford. The creativity is in deciding which of these a given feature and a given model require. The transformations themselves are mathematics, and once you see them that way, the folklore turns into a short list of consequences.

See you soon.