The Mathematics Behind Neural Networks Explained

The Keras workbook shows you how to build and train networks in three steps. This article opens the box and shows the mathematics those three steps are running. Everything a neural network does reduces to four operations: a weighted sum, a non-linear activation, a loss that measures error, and a gradient that says how to reduce it. We will build each in turn, then run a complete forward pass, loss, and weight update by hand.

The neuron: a weighted sum

The atom of a neural network is the neuron. It takes several inputs, multiplies each by a weight, adds a bias, and produces a single number.

z = \sum_{i=1}^{n} w_i x_i + b

Where the $x_i$ are the inputs, the $w_i$ are the weights the network learns, and $b$ is the bias, a learned offset.

The weights decide how much each input matters, and the bias shifts the result up or down independently of the inputs. On its own this is just linear regression. What makes a network more than a stack of linear regressions is the activation function applied next.

Activation functions

A neuron passes its weighted sum $z$ through a non-linear activation function $\phi$ to produce its output.

a = \phi(z)

The choice of $\phi$ is what gives networks their power, because a stack of purely linear layers collapses into a single linear layer.

The most common hidden-layer activation is the rectified linear unit, ReLU, which simply zeroes out negatives.

\text{ReLU}(z) = \max(0, z)

Its cheapness and its non-vanishing gradient for positive inputs are why it dominates hidden layers.

For a binary output, the sigmoid squashes any real number into the interval between 0 and 1, so it can be read as a probability.

\sigma(z) = \frac{1}{1 + e^{-z}}

For a multi-class output with $k$ classes, softmax turns a vector of $k$ scores into $k$ probabilities that sum to one.

\text{softmax}(z_j) = \frac{e^{z_j}}{\sum_{c=1}^{k} e^{z_c}}

These three activations map directly onto the output-layer rule from the workbook: no activation for regression, sigmoid for binary, softmax for multi-class.

A layer in matrix form

A layer is many neurons computed at once. Stacking the weights into a matrix $W$ and the biases into a vector $\mathbf{b}$, the whole layer is one matrix multiplication followed by an elementwise activation.

\mathbf{a} = \phi(W \mathbf{x} + \mathbf{b})

Where $\mathbf{x}$ is the input vector, $W$ has one row per neuron and one column per input, and $\phi$ is applied to each element.

This is exactly what layers.Dense(units) computes. If the input has $n$ features and the layer has $m$ units, then $W$ is an $m \times n$ matrix and $\mathbf{b}$ has length $m$, which gives the parameter count for that layer.

\text{parameters} = m \times n + m

The $m \times n$ term is the weights and the trailing $m$ is one bias per neuron, matching the counts you read off model.summary().

The forward pass

A network is a composition of layers. The output of one layer becomes the input to the next, so a two-hidden-layer network computes its prediction $\hat{y}$ as nested functions.

/math \hat{y} = \phi_3\big(W_3, \phi_2(W_2, \phi_1(W_1 \mathbf{x} + \mathbf{b}1) + \mathbf{b}2) + \mathbf{b}_3\big)

Where each $\phi$ is that layer’s activation and each $W, \mathbf{b}$ pair is its learned parameters.

Reading from the inside out is the forward pass: transform the input, apply an activation, feed it forward, and repeat until the output layer produces the prediction.

Loss functions

The loss measures how wrong a prediction is, as a single number the network tries to minimise. For regression, mean squared error averages the squared gap between predictions and targets over the $N$ examples.

\text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i – y_i)^2

Squaring makes every error positive and punishes large misses disproportionately.

For binary classification, binary cross-entropy measures the distance between the predicted probability $\hat{y}$ and the true label $y$, which is 0 or 1.

\text{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \big[, y_i \log \hat{y}_i + (1 – y_i) \log(1 – \hat{y}_i) ,\big]

When the true label is 1, only the first term survives and the loss is $-\log \hat{y}$, which is small when $\hat{y}$ is near 1 and large when it is near 0.

For multi-class classification with one-hot labels, categorical cross-entropy sums over the $k$ classes, keeping only the term for the correct class.

\text{CCE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{k} y_{i,c} \log \hat{y}_{i,c}

This is why the workbook pairs softmax with categorical cross-entropy: softmax produces the probabilities $\hat{y}_{i,c}$, and cross-entropy scores them.

Gradient descent: the learning rule

Training means adjusting every weight to reduce the loss. The gradient of the loss with respect to a weight, written $\frac{\partial L}{\partial w}$, is the direction in which the loss increases fastest, so we step in the opposite direction.

w \leftarrow w – \eta , \frac{\partial L}{\partial w}

Where $\eta$ is the learning rate, the size of each step.

Every parameter in the network is updated by this same rule on every step. The learning rate is the single most important hyperparameter: too large and the updates overshoot the minimum, too small and training crawls.

Backpropagation: the chain rule

The missing piece is how to compute $\frac{\partial L}{\partial w}$ for a weight buried deep in the network. Backpropagation is the chain rule applied to the composed layers, propagating the error backward from the loss. For a single weight feeding a neuron, the gradient factors into three local pieces.

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial w}

The first factor is how the loss changes with the neuron’s output, the second is the activation’s derivative, and the third, from $z = wx + b$, is simply the input $x$.

\frac{\partial z}{\partial w} = x

Because each layer’s gradient reuses the one computed for the layer above it, the whole network’s gradients are found in a single backward sweep. This is what loss.backward() does in PyTorch and what Keras runs inside model.fit.

Why the output layer and loss must match

The workbook’s central rule now has a mathematical reason. When you pair sigmoid output with binary cross-entropy, the derivative that starts backpropagation simplifies beautifully.

\frac{\partial L}{\partial z} = \hat{y} – y

The gradient is just the prediction minus the target, clean and well-scaled, which is why training is stable. The same tidy result holds for softmax with categorical cross-entropy, and for a linear output with mean squared error. Mismatch the pair, say sigmoid output with mean squared error, and this cancellation is lost, leaving a gradient that can shrink toward zero when the model is most wrong. That is the mathematics behind the most common beginner mistake.

A worked example, end to end

Take one neuron with two inputs, a sigmoid activation, and a single training example, and run one full step by hand. Let the inputs and target be

x_1 = 1, \quad x_2 = 2, \quad y = 1

and start with weights and bias

w_1 = 0.5, \quad w_2 = -0.5, \quad b = 0

Step one, the weighted sum.

z = (0.5)(1) + (-0.5)(2) + 0 = -0.5

Step two, the sigmoid activation gives the prediction.

\hat{y} = \sigma(-0.5) = \frac{1}{1 + e^{0.5}} \approx 0.378

Step three, the binary cross-entropy loss for a true label of 1.

L = -\log(0.378) \approx 0.973

Step four, the gradient at the output, using the clean sigmoid-with-cross-entropy result.

\frac{\partial L}{\partial z} = \hat{y} – y = 0.378 – 1 = -0.622

Step five, the gradient for each weight, multiplying by its input.

\frac{\partial L}{\partial w_1} = (-0.622)(1) = -0.622

\frac{\partial L}{\partial w_2} = (-0.622)(2) = -1.244

Step six, the weight update with a learning rate of $\eta = 0.1$.

w_1 \leftarrow 0.5 – (0.1)(-0.622) = 0.562

w_2 \leftarrow -0.5 – (0.1)(-1.244) = -0.376

Both weights moved in the direction that raises $z$, which raises $\hat{y}$ toward the target of 1. Run this loop thousands of times over many examples and the network learns, and every optimizer in Keras is doing exactly this arithmetic, just across millions of weights at once.

The whole picture

A neural network is four ideas stacked and repeated. A neuron computes a weighted sum. An activation bends it non-linearly. A loss scores the prediction. A gradient, found by the chain rule, says how to nudge every weight to make the score better. model.fit is a loop that runs the forward pass, computes the loss, backpropagates the gradients, and applies the update, over and over. Once you can see that loop in the equations, the Keras code stops being incantation and becomes a readable description of the mathematics you have just worked through by hand.

See you soon.

The Mathematics Behind Supervised Machine Learning