The Mathematics Behind Descriptive Statistics and EDA

Exploratory data analysis has a reputation as the informal part of the pipeline, the looking-around you do before the real modelling starts. That framing undersells it. Every summary statistic you compute during EDA is an estimator with a definition, an optimality property, and a breaking point, and knowing those three things is the difference between a summary that describes your data and one that quietly lies about it. The mean is not just “the average”; it is the unique minimiser of squared error, which is exactly why one outlier drags it away from the bulk. The median is not a cruder alternative; it is the minimiser of absolute error, which is exactly why it does not move. Correlation is not a general measure of relationship; it is a measure of one specific kind, and it is blind to every other kind by construction.

This article derives the descriptive toolkit from its definitions. It shows why the mean and median disagree on skewed data and what that disagreement measures, where the mysterious 1.5 in the outlier rule comes from, why variance divides by n-1 rather than n, and why a correlation of zero can hide a perfect relationship. The point is not to replace intuition but to arm it, so that when a histogram and a summary table disagree, you know which to trust.

Moments: the shape of a distribution in numbers

Nearly every descriptive statistic is a moment, a specific expectation that captures one aspect of a distribution’s shape. The raw moments are powers of the variable; the more useful central moments are powers of its deviation from the mean:

The index k selects the aspect. The first raw moment is the mean, locating the distribution. The second central moment is the variance, measuring spread. The third, once standardised, measures asymmetry. The fourth measures tail weight. Four numbers, four questions: where, how wide, how lopsided, how heavy-tailed. EDA is, to a first approximation, the practice of reading these four and then checking whether they tell the truth.

The mean, and why it is fragile

The sample mean is the familiar average:

Its defining property is not obvious from the formula but explains everything about its behaviour. The mean is the single number c that minimises the total squared distance to the data:

The proof is one derivative. Differentiate the sum with respect to c and set it to zero:

Solving gives $\sum x_i = nc$, so $c = \bar{x}$. The mean is where the squared-error summary is smallest, which is precisely why least-squares regression, whose whole objective is squared error, predicts conditional means.

This optimality is also the source of the mean’s fragility. Squaring means a point far from the centre contributes enormously to the objective, so the minimiser must move toward it to reduce that large squared term. Push one observation to infinity and the mean follows it to infinity. In the language of robust statistics, the mean has a breakdown point of zero: the fraction of the data you must corrupt to send the statistic arbitrarily far is $1/n$, which tends to zero. A single bad value ruins it. That is not a flaw to be patched; it is the direct consequence of minimising a squared loss, and it is why EDA never trusts a mean without also looking at a median.

Variance, and the n-1 that puzzles everyone

The second central moment is the variance, and its sample estimator carries a correction that trips up every newcomer:

Why n-1 and not n? The answer is that dividing by n produces a biased estimator, and the bias is derivable exactly. The intuition first: the deviations are measured from $\bar{x}$, the sample mean, not the true mean $\mu$. But $\bar{x}$ is itself the point that minimises the sum of squared deviations, so the deviations from it are systematically smaller than the deviations from the true mean would have been. Dividing by n would therefore underestimate the true variance.

The exact statement is that the sum of squared deviations has expectation:

so dividing by n-1 rather than n makes the estimator unbiased:

This is Bessel’s correction, and the deeper reading is about degrees of freedom. Estimating the mean from the data consumes one piece of information, since once $\bar{x}$ and any n-1 of the deviations are known, the last deviation is determined by the constraint that the deviations sum to zero:

Only n-1 of the deviations are free, so the sum of squares carries n-1 degrees of freedom, and dividing by the degrees of freedom rather than the count is what removes the bias. The same principle reappears everywhere a model estimates parameters before estimating a variance, which is why a regression with p coefficients divides its residual sum of squares by n-p.

The standard deviation is the square root of the variance:

and its only advantage over the variance is units. Variance is measured in the square of the feature’s units, which is uninterpretable; the standard deviation returns to the original units and can be read as a typical distance from the mean.

Skewness: the disagreement between mean and median

The third standardised moment measures asymmetry:

The cube is what makes it a signed tail detector. A value above the mean contributes a positive cube, one below contributes a negative cube, and because the contributions are cubed, values far out in a tail dominate. A distribution with a long right tail has a few large positive deviations whose cubes outweigh the many small negative ones, giving $\gamma_1 > 0$. A long left tail gives $\gamma_1 < 0$. Symmetric distributions have $\gamma_1 = 0$.

Skewness has a concrete signature in EDA, and it is the relationship between the mean and the median. For a right-skewed distribution the tail pulls the mean, which minimises squared error and therefore chases outliers, further out than the median, which does not. The ordering is reliable:

This is why comparing the mean and median is the fastest skewness check available. When they are close, the distribution is roughly symmetric and the mean is a fair summary. When the mean sits well above the median, the data is right-skewed, the mean is being dragged by a tail, and the median is the more honest measure of a typical value. Income data is the canonical case: mean income always exceeds median income, and the gap is a measure of the inequality in the tail. Reporting mean income alone overstates what a typical person earns, and the overstatement is exactly the skewness.

Kurtosis: the weight in the tails

The fourth standardised moment measures tail weight:

The fourth power punishes distance even more severely than the cube, and it is unsigned, so it responds to extreme values in either tail. The subtraction of 3 makes the normal distribution the reference point, since a normal has raw fourth standardised moment exactly 3; this shifted version is the excess kurtosis, zero for a normal.

The reading is about how much of the variance comes from rare extreme events versus routine moderate ones. Positive excess kurtosis means heavy tails: extreme values are more common than a normal would predict, and a large share of the variance is contributed by a handful of outliers. Negative excess kurtosis means light tails, with values clustered and few extremes. For EDA the practical use is a warning: high kurtosis signals that outliers are not anomalies to be deleted but a genuine feature of the process, and that any method assuming normal tails, which includes many default confidence intervals, will understate the frequency of extreme events. Financial returns are the standard example, notoriously heavy-tailed, and the 2008-style “twenty-sigma event that keeps happening” is simply kurtosis that a normal model refused to acknowledge.

Quantiles and order statistics: the robust alternative

Moments are one family of summaries; quantiles are the other, and they are built from a completely different operation. Sort the data into order statistics:

The q-quantile is the value below which a fraction q of the data lies. The median is the halfway quantile:

The median has its own optimality property, the exact analogue of the mean’s, and comparing the two derivations explains the entire robustness difference between them. The median is the minimiser of absolute error rather than squared error:

The reason it is robust falls out of differentiating this objective. The derivative of $|x_i – c|$ is the sign of $(x_i – c)$, which is +1 or -1 regardless of how far the point is. So the optimality condition balances the count of points above c against the count below, and moving an already-distant point further away does not change its sign and therefore does not move the median at all. Where the mean weights each point by its distance and can be dragged to infinity by one observation, the median weights each point by its sign and ignores magnitude entirely. Its breakdown point is 0.5: you must corrupt half the data before you can move the median arbitrarily, which is the highest breakdown point any location estimator can have.

The interquartile range applies the same robustness to spread. It is the distance between the 25th and 75th percentiles:

Because it depends only on the two quartiles and not on the extremes, the IQR is a robust measure of spread, unaffected by outliers in the way the standard deviation is not. The five-number summary that underlies every box plot, minimum, $Q_1$, median, $Q_3$, maximum, is entirely built from order statistics and is robust throughout, which is why a box plot survives contamination that would distort a mean-and-standard-deviation summary.

Where the 1.5 in the outlier rule comes from

The standard rule flags a point as an outlier if it lies beyond a fence set 1.5 interquartile ranges outside the quartiles:

The 1.5 looks arbitrary and is often taught as a convention, but it has a calibration against the normal distribution. For a normal, the quartiles sit at about $\pm 0.674\sigma$, so the IQR is about $1.35\sigma$. The upper fence is then:

so the fences sit at roughly $\pm 2.7\sigma$ from the centre. Under a normal distribution the probability of exceeding that is small:

about seven in a thousand. The 1.5 is chosen so that, for genuinely normal data, only around 0.7 percent of points are flagged, which is rare enough to mean “unusual” without flooding you with false alarms. Tukey’s wider fence at 3 IQRs, sitting near $4.7\sigma$, flags only the truly extreme. The rule is not a law of nature and a heavy-tailed distribution will trip it constantly, which is itself informative, but the number is a deliberate normal-calibrated trade-off, not a guess.

Correlation, and exactly what it can see

The workhorse bivariate summary is Pearson’s correlation, built from the covariance:

normalised by the two standard deviations:

That it always lies in [-1, 1] is not a definition but a theorem, and it is worth seeing why, because the proof reveals what correlation actually measures. The Cauchy-Schwarz inequality states that for any two random variables:

Apply it with $U = X – \mu_X$ and $V = Y – \mu_Y$:

Divide through and take square roots to get $|\rho| \le 1$. Crucially, Cauchy-Schwarz is an equality if and only if U and V are linearly dependent, meaning $Y – \mu_Y = a(X – \mu_X)$ for some constant a. So $\rho = \pm 1$ exactly when Y is an exact linear function of X, and values in between measure how close the relationship is to that straight line.

This is the whole content of the statistic, and its whole limitation. Correlation measures linear association and nothing else. A relationship can be perfect, deterministic, and obvious to the eye while having a correlation of zero, provided it is not linear. The standard demonstration is $Y = X^2$ on data symmetric about zero:

Every value of Y is exactly determined by X, yet the correlation is zero, because the parabola has no linear trend: as X increases, Y first falls then rises, and the two halves cancel. A correlation of zero does not mean “no relationship”, it means “no linear relationship”, and treating the two as the same is one of the most common errors in applied statistics.

Why you must plot, not just summarise

The limitation of correlation generalises into the central methodological rule of EDA: summary statistics are lossy, and different datasets can share every summary while looking completely different. Anscombe’s quartet is the famous construction, four datasets with the same mean of x, mean of y, variance of both, correlation, and fitted regression line, one a clean linear trend, one a curve, one a perfect line with a single outlier, one a vertical stack of points with one far-off leverage point. Every number in the summary table is identical; every scatter plot is unmistakably different.

The mathematical reason this is possible is that a handful of low-order moments cannot determine a distribution. The mean and variance fix two numbers; the correlation fixes one more; but the space of datasets consistent with those three numbers is vast, and it contains shapes that behave nothing alike under a model. The moments are projections of the data onto a few axes, and projections lose information by definition. Plotting is how you recover the dimensions the summary discarded, which is why EDA insists on the histogram, the scatter plot, and the box plot alongside the table, and why a summary statistic should never be the last word.

What the mathematics tells you to do

Report the median alongside the mean, always, because their difference is a free skewness diagnostic and because the mean has a breakdown point of zero while the median’s is one half. Prefer the IQR to the standard deviation when the data may be contaminated, since order statistics ignore the extremes that the squared deviations amplify. Read a mean sitting well above a median as right-skew, and summarise such data with the median. Treat high kurtosis as a statement that outliers are part of the process rather than errors to delete, and distrust any normal-based interval on heavy-tailed data. Remember the 1.5 fence is calibrated to flag about 0.7 percent of normal data, so on non-normal data it means something different. And never read a correlation without the scatter plot behind it, because $\rho = 0$ is consistent with a perfect nonlinear relationship and identical summaries are consistent with wildly different data.

The idea underneath all of it is that a descriptive statistic is a choice of loss function in disguise. The mean minimises squared error and inherits its sensitivity to outliers; the median minimises absolute error and inherits its robustness; the variance measures squared spread and the IQR measures ranked spread; correlation measures linear fit and sees nothing else. Each summary answers one question well and is silent, or misleading, on the rest. EDA done properly is not the casual prelude to analysis, it is the disciplined act of asking several summaries the same question and plotting the data when they disagree, because the moments are a shadow of the distribution and the shape is where the truth lives.

See you soon.