The Mathematics Behind CNNs and RNNs

A dense layer is the most general thing a neural network can do. Every input connects to every output, no assumption is made about how the inputs relate to each other, and the layer will learn any linear map you can write down. That generality is precisely the problem. Fed an image, a dense layer does not know that neighbouring pixels belong together. Fed a sentence, it does not know that word five came after word four. It sees a flat vector of numbers whose ordering is arbitrary, and it must learn from data what the structure of the problem could have told it for free.

Convolutional and recurrent networks are two answers to that single complaint, and mathematically they are closer relatives than their names suggest. Both take a structured index, space for images and time for sequences, and both reuse the same weights at every position along it. The difference is what they do with the shared weights: a CNN applies them in parallel across space, while an RNN applies them in a loop across time. Everything else, including the counting argument that makes CNNs tractable and the gradient product that makes RNNs fail, follows from that choice.

What a dense layer costs

Begin with the counting argument, because it settles the matter before any theory. A modest colour image of 224 by 224 pixels has:

Connect that to a hidden layer of 1,000 units and the weight matrix alone holds:

One hundred and fifty million weights, for one layer, before the network has learned anything at all. Worse, those parameters are spent badly. The weight connecting pixel (14, 22) to a hidden unit is entirely separate from the weight connecting pixel (15, 22), so a model that has learned to detect an edge in one corner of the image has learned nothing whatsoever about edges anywhere else. Every position must be taught independently, and the training data required grows accordingly.

The structural facts the dense layer ignores are simple and true: nearby pixels are related, distant ones mostly are not, and a pattern that matters in one place matters in another. Convolution encodes exactly those facts.

The convolution operation

Take a small array of weights, the kernel, and slide it across the input, computing a dot product at each position. In one dimension:

In two dimensions, the same idea with a square kernel:

A pedantic note that matters when reading papers: the true mathematical convolution flips the kernel, using $x[i – m]$ rather than $x[i + m]$. What every deep learning framework actually computes is cross-correlation, without the flip. Since the kernel weights are learned rather than given, the flip is irrelevant in practice, because whichever orientation is useful is the one gradient descent will find. The name stuck; the flip did not.

Two properties fall directly out of the formula. First, locality: $y[i, j]$ depends only on a $k \times k$ patch of the input, so the layer cannot mix distant pixels in a single step. Second, parameter sharing: the same $w$ appears in the expression for every output position. The kernel is not a weight per pixel, it is a weight per offset within the patch, and it is reused everywhere.

Equivariance is a theorem

Parameter sharing has a precise consequence. Let $T$ be a translation operator that shifts an input by some amount. Then convolution satisfies:

Shift the image, and the feature map shifts identically. This is translation equivariance, and it is not an empirical tendency of trained networks but an algebraic property of the operation itself, true at initialisation and true forever. It is the mathematical statement of the intuition that an edge detector should detect an edge wherever the edge happens to be, and it is exactly what the dense layer could not express.

Counting the parameters again

Real convolutional layers operate on multiple input channels and produce multiple output channels. The full expression, with stride s, is:

Each output channel is a separate kernel spanning all input channels, which is how a layer learns several different detectors at once. The parameter count is:

Now return to the 224 by 224 image. A layer with 64 filters of size 3 by 3 over 3 input channels costs:

Against 150 million. The count is independent of the image’s size entirely, since it depends only on kernel dimensions and channel counts, so the same layer processes a 224-pixel image and a 1,024-pixel image with identical weights. That independence is the practical gift of parameter sharing, and it is why convolutional networks were feasible on hardware that could never have trained the dense equivalent.

Output shapes and the receptive field

The arithmetic of shapes is worth having memorised, since it is the source of most shape errors in practice. For an input of size n, kernel k, padding p, and stride s:

The floor is where information is silently discarded when the stride does not divide evenly. Setting $p = (k-1)/2$ with $s = 1$ and odd k gives o = n, the “same” padding that preserves spatial size and is why odd kernels dominate.

Locality raises an obvious objection: if each layer sees only a $k \times k$ patch, how can a network recognise an object spanning half the image? The answer is composition. Define the receptive field $r_L$ as the region of the original input influencing one unit at layer L. It grows according to:

The product of strides is the key term. With stride 1 everywhere, receptive fields grow additively and slowly, a few pixels per layer. Introduce stride-2 layers and the multiplier doubles at each one, so the growth becomes geometric and a dozen layers can see the entire image. Depth is not decoration; it is the mechanism by which local operations accumulate global scope, and each layer composes patterns found by the layer beneath it.

Pooling

Max pooling takes a window and keeps only its largest value:

It has no parameters and it does two things. It shrinks the spatial dimensions, which cuts computation and accelerates receptive field growth. And it converts equivariance into approximate invariance: if a feature moves by less than the pooling window, the maximum is unchanged, so the output does not care exactly where within the window the feature sat. Convolution says “the same pattern matters everywhere”; pooling says “and its exact position is not important”. Both are structural knowledge injected by the architecture rather than learned from data.

The gradient through max pooling routes entirely to whichever input was the maximum and sends zero to the others, which is why pooling layers are sometimes described as making the network’s credit assignment sparse.

The recurrent network

Sequences have the same difficulty in a different index. A sentence of varying length cannot feed a fixed-size dense layer, and word order carries meaning that a flat vector loses. The recurrent answer applies one shared weight matrix repeatedly along the time axis, carrying a hidden state forward:

The parallel with convolution is exact in spirit. The CNN reuses one kernel at every spatial position; the RNN reuses $W_{hh}$ and $W_{xh}$ at every time step. Sharing again buys independence from input size, so a single set of weights handles a sequence of any length, and $h_t$ is a summary of everything seen so far, in principle a memory of unbounded reach.

In principle. The gradient has other ideas.

Backpropagation through time, and why it fails

To train the network, the loss at a late step must send credit back to early steps. Unrolling the recurrence and applying the chain rule gives a product of Jacobians:

and each factor in that product, differentiating the recurrence, is:

The product is the problem. Take norms and apply the submultiplicative inequality:

where $\gamma_{\phi}$ bounds the activation derivative and $\gamma_W$ is the largest singular value of $W_{hh}$. The exponent is the distance in time. Whatever the base, an exponential in T has only two behaviours:

Vanishing or exploding, with no comfortable middle, because balancing on exactly 1 across every step and every input is not something training can maintain. For tanh, $\gamma_{\phi} \le 1$, so vanishing is guaranteed whenever the largest singular value of $W_{hh}$ is below 1. For the sigmoid, $\gamma_{\phi} \le 1/4$, which makes matters four times worse per step.

Put numbers on it. With a modest factor of 0.8 per step, credit reaching back 50 steps is scaled by:

The early words in a long sentence receive essentially no gradient. The network is not choosing to forget them; the arithmetic has erased their influence on the update. Exploding gradients are the milder failure, since they announce themselves as NaNs and are treatable by clipping the gradient norm. Vanishing gradients are silent: the model trains, the loss falls, and long-range dependencies are simply never learned.

The root cause is worth naming precisely. It is not depth as such, it is that the recurrence multiplies by the same matrix at every step. Repeated multiplication is exponential, and exponentials are unforgiving.

The LSTM: replacing multiplication with addition

The LSTM’s fix is structural. It introduces a cell state $c_t$ alongside the hidden state and gives it an update rule that is additive rather than multiplicative. Three gates, each a sigmoid producing values between 0 and 1, control the flow:

A candidate contribution is computed with tanh:

and the cell state updates as a gated sum, where $\odot$ is elementwise multiplication:

The gates have readable jobs. The forget gate decides what fraction of the old cell state survives, the input gate decides how much of the new candidate enters, and the output gate decides how much of the cell is exposed as the hidden state. But the mathematics that matters is in the shape of the cell update. Differentiate it along the direct path:

and the gradient across many steps travels along:

Compare this with the vanilla RNN’s product of $\operatorname{diag}(\phi'(z_s)) W_{hh}$. The weight matrix is gone from the direct path, and the repeated factor is the forget gate. When the network learns $f_s \approx 1$ for some dimension, the product along that dimension is approximately 1 no matter how many steps it spans, and the gradient arrives at step t essentially undiminished. This is the constant error carousel: an uninterrupted highway along which credit flows without exponential decay.

The crucial difference is that $f_s$ is learned and input-dependent, so the network chooses which dimensions to hold open and for how long, rather than suffering a fixed decay rate imposed by the spectral norm of a shared matrix. It also explains a practical trick with a mathematical justification: initialising the forget gate bias $b_f$ to a positive value pushes $f_s$ near 1 at the start of training, opening the highway before the model has learned anything, so long-range gradients survive the early epochs during which the model would otherwise have learned to ignore the distant past.

Note finally that the LSTM does not repeal the bound derived earlier. Gradients still flow through $W$ on the other paths, and they still decay there. What the architecture provides is one additive route that escapes the exponential, which is enough, and it is the same insight that residual connections later brought to very deep feedforward networks: when a product of Jacobians is the enemy, add a path whose Jacobian is the identity.

What the mathematics tells you to do

The rules of thumb are consequences. Use odd kernels with $p = (k-1)/2$ when you need to preserve spatial size, because that is what the shape formula returns. Expect stride to matter more than kernel size for coverage, because strides enter the receptive field formula as a product while kernels enter as a sum. Do not be surprised that a convolutional layer has a tiny parameter count independent of image size, since $k \cdot k \cdot C_{\text{in}} \cdot C_{\text{out}}$ contains no n. Clip gradient norms when training recurrent networks, because explosion is the failure mode you can see and clip is the direct treatment. Reach for an LSTM or GRU rather than a vanilla RNN whenever dependencies exceed a handful of steps, because $\left(\gamma_\phi \gamma_W\right)^{T-t}$ has already decided the outcome. And initialise the forget gate bias positive, because opening the carousel early is free.

The unifying idea is that both architectures are structure imposed by hand rather than discovered from data. The CNN asserts that the same pattern matters at every location and gets equivariance as a theorem and a hundred-thousand-fold reduction in parameters as a corollary. The RNN asserts that the same transition matters at every time step, gets sequence-length independence in return, and pays for it with an exponential in the gradient that the LSTM’s additive cell path was invented to defuse. Architecture, seen mathematically, is nothing more than the assumptions you were willing to make before the data arrived.

See you soon, Andrei.