The Mathematics Behind Supervised Machine Learning

The scikit-learn guide teaches a workflow: split the data, fit a model, evaluate it, and improve it. Underneath that uniform fit and predict interface, each model is solving a specific mathematical problem, and each metric is a specific formula. This article works through both. You will see why linear regression has a closed-form solution, why lasso performs feature selection and ridge does not, why logistic regression uses log-loss rather than squared error, and exactly how precision, recall, and the ROC curve are computed.

The supervised learning problem

Supervised learning has one goal: find a function $f$ that maps inputs to outputs, using labelled examples. Given $N$ training pairs of features $\mathbf{x}_i$ and labels $y_i$, we search for the f that minimises the average loss between predictions and truth.

Where L is a loss function measuring the gap between a prediction and its label.

Every model in the article is a different choice of the family for f and the loss L. Regression uses a continuous loss like squared error; classification uses a loss suited to probabilities. The fit method is this minimisation, and predict evaluates the learned f on new inputs.

K-nearest neighbours: distance and vote

KNN has no training in the usual sense; it stores the data and classifies a new point by the labels of its nearest neighbours. Nearness is measured by the Euclidean distance between two feature vectors.

Where the sum runs over the p features.

To classify a new point, KNN finds the k training points with the smallest distance and takes a majority vote. Suppose k = 3 and a new point’s three nearest neighbours have labels 1, 1, and 0. The prediction is the majority.

Because distance sums over features directly, a feature measured in thousands will dominate one measured in fractions, which is exactly why KNN requires the feature scaling of a later section.

Linear regression: least squares

Linear regression models the target as a weighted sum of the features plus an intercept.

The coefficients $\beta$ are chosen to minimise the residual sum of squares, the total squared gap between predictions and targets.

Writing the features as a matrix X (with a column of ones for the intercept) and the targets as a vector $\mathbf{y}$, this minimisation has a closed-form solution, the normal equation.

This is why linear regression needs no iterative training: the optimal coefficients are computed directly in one matrix expression.

As a worked example, fit a line $\hat{y} = \beta_0 + \beta_1 x$ to the three points $(1, 2)$, (2, 2), and (3, 4)

. The least-squares slope is the covariance of x and y over the variance of x.

The intercept then places the line through the means.

So the fitted line is $\hat{y} = 0.67 + x$, the line that makes the total squared vertical error as small as possible.

Ridge and lasso: regularisation

Regularisation adds a penalty on the size of the coefficients, discouraging the model from fitting noise. Ridge regression adds the squared size of the coefficients, the L2 penalty, controlled by a strength $\alpha$.

The larger $\alpha$ is, the more the coefficients are shrunk toward zero, which reduces variance at the cost of a little bias.

Lasso regression instead penalises the absolute size of the coefficients, the L1 penalty.

The two penalties behave very differently near zero. The squared L2 penalty has a vanishing gradient as a coefficient approaches zero, so ridge shrinks coefficients but rarely sets them exactly to zero. The absolute-value L1 penalty has a constant gradient right up to zero, so lasso can drive coefficients exactly to zero, removing those features entirely. That is the mathematical reason lasso performs feature selection and ridge does not.

Logistic regression: probabilities and log-loss

For binary classification, logistic regression first computes a linear score, then passes it through the sigmoid to produce a probability between 0 and 1.

The class boundary is where the probability crosses one half, which is where the linear score is zero.

The model is trained not with squared error but with log-loss, also called binary cross-entropy, which measures the distance between the predicted probability and the true label.

Squared error is avoided here because, combined with the sigmoid, it produces a non-convex loss surface full of local minima. Log-loss combined with the sigmoid is convex, so gradient descent reliably finds the single best solution. Unlike linear regression, there is no closed form, so the coefficients are found by iterative optimisation.

Feature scaling

Any method that measures distance or penalises coefficient size is sensitive to the scale of the features. Standardisation rescales each feature to have mean zero and standard deviation one.

Where $\mu$ and $\sigma$ are the feature’s mean and standard deviation on the training data.

After this transform, a feature in kilograms and a feature in millimetres contribute comparably to a distance or a penalty. This is why KNN, ridge, and lasso all need scaling, and why it belongs inside a pipeline so the statistics are computed only on the training fold.

Regression metrics: RMSE and R-squared

Once a regression model is fit, two numbers describe its quality. Root mean squared error is the typical size of a prediction error, in the units of the target.

The coefficient of determination, $R^2$, expresses how much of the target’s variance the model explains, comparing its error to the error of simply predicting the mean.

An $R^2$ of 1 is a perfect fit, 0 means the model does no better than predicting the average, and negative values mean it does worse.

Classification metrics: precision, recall, F1

Accuracy alone misleads on imbalanced classes, so classification is judged from the confusion matrix of true positives, false positives, true negatives, and false negatives. Precision asks: of the cases predicted positive, how many really were.

Recall asks the complementary question: of the actual positives, how many did the model find.

The F1 score is their harmonic mean, high only when both are high.

As a worked example, suppose a model produces $TP = 40$, $FP = 10$, $FN = 20$, and $TN = 130$. Precision is $40 / 50 = 0.80$ and recall is $40 / 60 = 0.67$, so the F1 score is

Accuracy here would be $170 / 200 = 0.85$, comfortably higher than F1, which is exactly how accuracy flatters a model that misses a third of the positives.

The ROC curve and AUC

A probabilistic classifier does not commit to a threshold. The ROC curve traces performance across all thresholds, plotting the true positive rate against the false positive rate.

As the threshold sweeps from 1 down to 0, each setting produces one point, and the curve connects them.

The area under this curve, the AUC, summarises the whole curve as one number. It has a clean interpretation: the probability that the model ranks a random positive example above a random negative one. An AUC of 1 is perfect ranking, and 0.5 is no better than a coin flip, which is why AUC is the metric of choice when classes are imbalanced and a single threshold would hide the model’s behaviour.

Overfitting as a bias-variance trade-off

The reason regularisation and cross-validation exist is captured in one decomposition. The expected error of a model splits into three parts.

Bias is error from a model too simple to capture the pattern, and variance is error from a model so flexible it fits the noise.

A high-degree model or a KNN with $k = 1$ has low bias but high variance, memorising the training data and generalising poorly. A heavily regularised model has higher bias but lower variance. Tuning $\alpha$, or $k$, or the depth of a tree is the search for the point where their sum is smallest, and cross-validation is how you estimate that sum honestly without touching the test set.

The whole picture

Every model in the article is the same template with different choices. KNN chooses a distance and votes. Linear regression chooses a linear function and squared-error loss, solved in closed form. Ridge and lasso add an L2 or L1 penalty, and the shape of that penalty near zero decides whether features are shrunk or dropped. Logistic regression swaps in the sigmoid and log-loss to output convex, well-behaved probabilities. The metrics then measure what the loss could not fully capture, with precision, recall, F1, and AUC guarding against the imbalance that accuracy hides. Seen this way, scikit-learn’s uniform fit and predict interface is a clean surface over a handful of precise optimisation problems, and knowing the mathematics tells you which model, penalty, and metric each situation actually calls for.

See you soon.