The Mathematics Behind Linear Classifiers: Margins and the SVM

Logistic regression and the support vector machine are usually taught as different animals. One outputs probabilities and comes from statistics; the other draws a widest-possible street between the classes and comes from optimisation theory. In scikit-learn they even live in different modules. Yet if you write down what each one actually computes, the difference collapses to a single line, and everything else, including the entire vocabulary of margins, support vectors, and the mysterious C parameter, follows from that one line.

The shared skeleton is this. Both models score a point with the same expression:

and both predict by sign. The prediction is class +1 when $f(x) > 0$ and class -1 when $f(x) < 0$. That is the whole model. Two classifiers, one function, identical at prediction time. What separates them is not the hypothesis but the loss: the rule by which they decide, during training, which of the infinitely many valid $(w, b)$ pairs deserves to be chosen. This article derives that choice, and shows what the geometry of the chosen boundary means.

The geometry of a hyperplane

The decision boundary is the set where the score vanishes:

Two facts about this set matter, and both are one-line proofs. First, w is perpendicular to the boundary. Take any two points $x_1$ and $x_2$ lying on it. Both satisfy the equation, so subtracting gives:

The vector $x_1 – x_2$ lies along the boundary, and its inner product with w is zero, so w is normal to the hyperplane. The weight vector is not an abstract parameter list, it is a direction in the feature space, pointing perpendicular to the decision surface, and it is the direction along which the score changes fastest.

Second, the score measures distance. The perpendicular distance from any point $x_0$ to the boundary is:

So $f(x)$ is not an arbitrary number. Divided by $\lVert w \rVert$, it is literally how far the point sits from the boundary, and its sign says which side. Every quantity in what follows is a variation on this formula.

Functional and geometric margins

Encode the labels as $y_i \in {-1, +1}$, a convention chosen precisely because it lets correctness and distance combine into one product. Define the functional margin of a training point:

Read the sign: when the label is +1 and the score is positive, the product is positive; when the label is -1 and the score is negative, the product is positive again. So $\hat{\gamma}_i > 0$ means correct, $\hat{\gamma}_i < 0$ means wrong, and the magnitude says how emphatically. One number carries both facts.

The functional margin has a defect, though, and the defect is instructive. Multiply both w and b by any positive constant c. The boundary ${x : c,w^{\top}x + c,b = 0}$ is exactly the same set of points, so the classifier is unchanged, yet every functional margin has been multiplied by c. Confidence can be inflated arbitrarily without improving anything. The cure is to divide out the scale:

This is the geometric margin, and comparing it with the distance formula above shows what it is: the signed perpendicular distance from the point to the boundary, positive when classified correctly. Unlike its functional cousin, it is invariant to rescaling and therefore means something.

Maximising the margin

Among all boundaries that separate the classes, which is best? The support vector machine’s answer is the one that stays furthest from the closest points on either side. Define the margin of the whole dataset as the smallest geometric margin:

and ask for the boundary maximising it:

As written this is unpleasant, containing a minimum inside a maximum and a ratio. The scale invariance we just complained about now becomes the tool that cleans it up. Since multiplying $(w, b)$ by any constant leaves the boundary and all geometric margins untouched, we are free to fix the scale however we like. Choose the normalisation that sets the smallest functional margin to exactly 1:

This is not an assumption about the data. It is a choice of units, available for free because the ratio was invariant. With it, the dataset’s geometric margin becomes:

and the margin band, the empty street between the two classes, has width $2/\lVert w \rVert$. Maximising that width means minimising $\lVert w \rVert$, and since squaring is monotone on non-negative numbers, we can minimise a convenient quadratic instead. The hard-margin SVM is:

This is worth pausing on, because the transformation is the mathematical heart of the method. A geometric wish, “put the widest possible street between the classes”, has become a convex quadratic programme with linear constraints: a problem class with a unique global optimum and reliable solvers. No local minima, no restarts, no initialisation luck. The awkward maximin over a ratio and the tidy quadratic are the same problem wearing different clothes.

Note also what $\lVert w \rVert$ now means. It looks like a regularisation penalty, and it behaves like one, but it arrived here as the reciprocal of the margin. Shrinking the weights and widening the street are the same act.

Why only some points matter

The constrained problem has a Lagrangian with one multiplier $\alpha_i \ge 0$ per training point, and the Karush-Kuhn-Tucker conditions include complementary slackness:

The condition is a product forced to zero, so for each point at least one factor must vanish. If the point sits strictly outside the margin, its bracket is strictly positive, forcing $\alpha_i = 0$. Only points sitting exactly on the margin boundary, where the bracket is zero, may have $\alpha_i > 0$.

The stationarity condition then expresses the solution as a weighted sum of training points:

Combine the two. Every point with $\alpha_i = 0$ contributes nothing to w, so the fitted boundary depends only on the handful of points touching the margin. Those are the support vectors, and the name is now literal rather than metaphorical: they hold the street open, and everything else is irrelevant. Delete a thousand comfortably classified points and refit, and the boundary does not move by a millimetre. Move one support vector and the whole thing swings.

This is a genuine difference from logistic regression, where every point contributes a nonzero gradient forever and distant correct points continue to nudge the boundary. It also explains the SVM’s characteristic robustness and its characteristic fragility in one sentence: it ignores the bulk of the data and stakes everything on the boundary cases.

Soft margins, and where hinge loss comes from

Real data is not separable, and the hard-margin constraints then have no feasible solution at all. Introduce a slack variable $\xi_i \ge 0$ per point, permitting violation at a price:

Now watch the slack variables disappear. At the optimum the solver will never pay more than it must, so each $\xi_i$ takes the smallest value satisfying its constraint. Rearranged, the constraint demands $\xi_i \ge 1 – y_i f(x_i)$, and combined with $\xi_i \ge 0$ the minimum feasible value is:

Substituting back eliminates the constraints entirely and leaves an unconstrained objective:

The second term is the hinge loss, and here it is not a definition handed down but a derived quantity: it is exactly the cost of violating the margin constraints, read off from the geometry. Writing $m_i = y_i f(x_i)$ for the functional margin, the hinge is:

Its shape encodes the SVM’s whole attitude. Beyond $m = 1$ the loss is flat at exactly zero, so a point comfortably on the right side contributes nothing, which is complementary slackness reappearing as a picture. Below $m = 1$ the penalty grows linearly, and the kink at 1 is what makes the solution sparse in support vectors.

Hinge against logistic

Logistic regression optimises a different loss on the same margin variable:

Both are convex surrogates standing in for the loss we actually care about, the zero-one loss:

which counts mistakes and is useless for optimisation, being flat everywhere with a discontinuity at zero. It has no gradient to follow, and minimising it exactly is NP-hard. Hinge and logistic are both convex upper bounds on it, which is the entire reason either exists.

The difference between them lives at $m > 1$. The hinge is identically zero there; the logistic loss is small but strictly positive, since $\log(1 + e^{-m})$ never reaches zero for finite m. Differentiating shows the consequence:

The factor $\sigma(-m_i)$ decays but never vanishes, so every point, however confidently correct, keeps pulling on the boundary forever, with a weight that decays smoothly with confidence. The hinge’s gradient, by contrast, is exactly zero past the margin and $-y_i x_i$ inside it. One model listens to everyone quietly; the other listens only to the points near the line and ignores the rest completely.

That single difference generates everything else people say about these models. Logistic regression yields calibrated probabilities through $\sigma(f(x))$, because its loss is the negative log-likelihood of a Bernoulli model and is proper for that purpose. The SVM yields no probabilities, because its loss makes no likelihood claim, which is why scikit-learn must bolt on Platt scaling to fake them. The SVM is sparse in support vectors; logistic regression is not sparse in anything. Neither is a defect. They are two positions on how much attention distant points deserve.

What C actually is

Divide the soft-margin objective by C and it becomes:

Compare it with the standard regularised form $\sum_i L_i + \lambda \lVert w \rVert^2$ and read off the correspondence:

C is the reciprocal of the regularisation strength, and this is the origin of the direction that trips people up in scikit-learn, where large C means less regularisation. The mathematics makes it obvious rather than arbitrary. Large C means violations are expensive, so the optimiser will contort the boundary to avoid them, tolerating a large $\lVert w \rVert$ and therefore a narrow margin: low bias, high variance. Small C means violations are cheap, so the optimiser prefers a small $\lVert w \rVert$, a wide margin, and a boundary that accepts some errors in exchange for stability: high bias, low variance.

The identity $\gamma = 1/\lVert w \rVert$ ties the knot. Tuning C is tuning the margin width, and tuning the margin width is tuning the bias-variance trade-off. Three descriptions, one dial. The same reading applies to C in scikit-learn’s LogisticRegression, which uses the identical convention, so a grid over C on a log scale is a grid over margin widths.

The dual, and the door to kernels

Solving through the Lagrangian yields the dual problem, stated entirely in terms of the multipliers:

The upper bound $\alpha_i \le C$ is where the slack penalty ends up, which is a neat confirmation that C caps how much influence any single point may exert: no individual training example, however badly placed, can pull the boundary further than C allows. Outliers are structurally limited.

The dual’s more famous consequence is the appearance of the data only through inner products $x_i^{\top}x_j$. Nowhere does the optimisation need the individual vectors, only their pairwise products. Substituting any valid kernel $K(x_i, x_j)$ for that inner product therefore fits a linear boundary in a transformed space that may be far higher-dimensional, without ever computing the transformation. That is the kernel trick, and it exists because of an accident of algebra in the dual form, not because anyone designed it in.

What the mathematics tells you to do

Scale your features before fitting either model, because both objectives penalise $\lVert w \rVert$, and a feature measured in thousands needs a tiny weight to have any effect while a feature measured in units needs a large one. The penalty does not know the difference and simply crushes whichever weight it can, so the regularisation you thought you were applying to complexity is applied to units instead.

Search C on a logarithmic grid, because the quantity it controls, $1/\lVert w \rVert$, is a scale rather than an offset, and equal ratios matter rather than equal differences. Expect large C to overfit narrow margins and small C to underfit wide ones, and read a chosen C as a statement about how much margin violation your data deserves.

Prefer logistic regression when you need probabilities, since the SVM’s loss makes no probabilistic claim and any probabilities you extract are a post-hoc fit. Prefer the SVM when the decision boundary is what matters and the bulk of the data is uninformative, since the hinge’s flat region will discard it for free.

The deeper point is the one the shared skeleton makes. These models are not different theories of classification. They are the same linear score under two different opinions about what a training point owes you once it is already on the right side of the line. Logistic regression says: keep contributing, quietly, forever. The SVM says: past the margin, be silent. Every practical difference between them, sparsity, calibration, robustness, and the meaning of C, is that one disagreement, worked out to its conclusions.

Hope this helps, Andrei.