PyTorch Deep Learning Fundamentals: 10 Code-Along Examples

The PyTorch article promise is that deep learning is a workflow that is the same every single time: tensors in, layers stacked, a loss measured, gradients computed, weights nudged, repeat. This workbook builds that workflow piece by piece, so that by Example 8 you are reading the full training loop as a familiar ritual rather than a wall of code. Run them in order; each one uses what came before.

1. Tensors: the foundation

A tensor is a NumPy array with two superpowers: it can live on a GPU, and it can remember the operations performed on it. Creation and the key attributes come first.

import torch
readings = torch.tensor([[1.0, 2.0], [3.0, 4.0]])   # from a Python list
zeros = torch.zeros(2, 3)                             # 2x3 of zeros
noise = torch.randn(2, 2)                             # random normal values
print(readings.shape)    # torch.Size([2, 2])
print(readings.dtype)    # torch.float32
print(readings.device)   # cpu
# operations look like NumPy
print(readings + noise)         # elementwise add
print(readings @ readings)      # matrix multiply
print(readings.mean(), readings.sum())

.shape, .dtype, and .device are the three attributes you will check constantly when debugging, because most PyTorch errors are a shape mismatch, a type mismatch, or two tensors on different devices. The operations themselves read exactly like NumPy.

2. Layers and nn.Sequential

nn.Linear is a fully connected layer: a matrix of weights plus a bias vector. nn.Sequential pipes data through a stack of layers in order, which is enough to define a real network.

import torch
import torch.nn as nn
model = nn.Sequential(
    nn.Linear(4, 8),   # 4 inputs -> 8 hidden units
    nn.ReLU(),
    nn.Linear(8, 2)    # 8 hidden -> 2 outputs
)
sample = torch.randn(1, 4)        # one row with 4 features
output = model(sample)
print(output)                     # two raw scores
# count the parameters
total = sum(p.numel() for p in model.parameters())
print(f"Parameters: {total}")     # (4*8 + 8) + (8*2 + 2) = 58

The parameter count is worth doing by hand once: each Linear holds inputs × outputs weights plus one bias per output, so this little network has 58 learnable numbers. Training is nothing more than finding good values for them.

3. Activation functions

Without activations, stacked linear layers collapse into one big linear layer. Each activation has a job: sigmoid squashes to a probability, softmax makes scores sum to one, ReLU keeps hidden layers non-linear and fast.

import torch
import torch.nn as nn
scores = torch.tensor([-2.0, 0.0, 3.0])
print(nn.Sigmoid()(scores))     # [0.119, 0.500, 0.953]  each in (0, 1)
print(nn.Softmax(dim=-1)(scores))  # [0.006, 0.047, 0.946]  sums to 1
print(nn.ReLU()(scores))        # [0.0, 0.0, 3.0]  negatives become zero
print(nn.LeakyReLU(0.1)(scores))   # [-0.2, 0.0, 3.0]  a trickle survives

Read the outputs against the inputs. Sigmoid turns any number into something probability-like, softmax turns a set of scores into a distribution, and ReLU simply cuts negatives to zero. Leaky ReLU lets a small negative trickle through, which prevents the “dying ReLU” problem where a neuron gets stuck outputting zero forever.

4. Loss functions

The loss is a single number measuring how wrong the model is. Regression uses mean squared error; multi-class classification uses cross-entropy. One critical convention: CrossEntropyLoss applies softmax internally, so your model must output raw scores.

import torch
import torch.nn as nn
# regression: MSE
prediction = torch.tensor([2.5, 0.0, 2.1])
target = torch.tensor([3.0, -0.5, 2.0])
mse = nn.MSELoss()
print(mse(prediction, target))   # mean of squared differences
# multi-class: CrossEntropy on RAW scores (logits)
logits = torch.tensor([[2.0, 0.5, -1.0]])   # no softmax applied!
label = torch.tensor([0])                    # the true class index
ce = nn.CrossEntropyLoss()
print(ce(logits, label))

The comment is the lesson: do not add a softmax layer to a model trained with CrossEntropyLoss, because the loss applies it internally and doubling it up quietly hurts training. This is one of the most common beginner mistakes in PyTorch.

5. Autograd: where gradients come from

Set requires_grad=True and a tensor remembers every operation done to it. Calling .backward() on a result walks that history in reverse, filling each tensor’s .grad with its gradient.

import torch
w = torch.tensor(2.0, requires_grad=True)   # a "weight" we want to tune
x = torch.tensor(3.0)                        # an input
y = w * x            # forward: y = 6
loss = (y - 12) ** 2  # we wanted 12, we got 6, loss = 36
loss.backward()       # compute d(loss)/d(w) automatically
print(loss.item())    # 36.0
print(w.grad)         # -36.0: increasing w decreases the loss

The gradient of -36 says “the loss falls if you increase w,” which points exactly toward the right answer (w = 4 would give y = 12). Every deep network does precisely this, just across millions of weights at once, and loss.backward() is the line that makes it happen.

6. The optimizer ritual

An optimizer uses those gradients to nudge the weights. The three-line ritual, zero the gradients, backward, step, is the heartbeat of all PyTorch training. Watch it reduce a loss in one step.

import torch
import torch.nn as nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()
features = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[5.0]])
for step in range(3):
    optimizer.zero_grad()               # 1. clear old gradients
    prediction = model(features)        # 2. forward pass
    loss = criterion(prediction, target)
    loss.backward()                     # 3. compute gradients
    optimizer.step()                    # 4. nudge the weights
    print(f"step {step}: loss {loss.item():.4f}")
# loss shrinks every step

zero_grad matters because PyTorch accumulates gradients by default; forget it and each step uses stale gradients from previous passes. SGD is the baseline optimizer; swap in torch.optim.Adam(model.parameters(), lr=0.01) and nothing else changes, which is why Adam is the robust default.

7. Datasets and DataLoaders

Real training runs on batches, not single rows. TensorDataset zips features and labels together, and DataLoader serves them up in shuffled batches.

import torch
from torch.utils.data import TensorDataset, DataLoader
features = torch.randn(100, 4)                  # 100 samples, 4 features
labels = torch.randint(0, 2, (100,))            # 100 binary labels
dataset = TensorDataset(features, labels)
loader = DataLoader(dataset, batch_size=32, shuffle=True)
for batch_features, batch_labels in loader:
    print(batch_features.shape, batch_labels.shape)
# torch.Size([32, 4]) torch.Size([32])  x3, then a final partial batch of 4

Shuffling matters for training, so the model never sees the data in a memorisable order, and is switched off for validation so results are reproducible. Batch sizes of 32, 64, or 128 are the common defaults.

8. The full training loop

Everything assembles here. Synthetic data with a real learnable pattern, a small network, Adam, and the ritual from Example 6 wrapped in epochs and batches. Watch the loss fall.

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
# synthetic two-class data: two clouds of points
torch.manual_seed(0)
class_a = torch.randn(100, 2) + torch.tensor([2.0, 2.0])
class_b = torch.randn(100, 2) + torch.tensor([-2.0, -2.0])
features = torch.cat([class_a, class_b])
labels = torch.cat([torch.zeros(100), torch.ones(100)]).long()
loader = DataLoader(TensorDataset(features, labels),
                    batch_size=32, shuffle=True)
model = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 2))
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(5):
    epoch_loss = 0
    for batch_features, batch_labels in loader:
        optimizer.zero_grad()
        prediction = model(batch_features)
        loss = criterion(prediction, batch_labels)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
    print(f"epoch {epoch}: loss {epoch_loss / len(loader):.4f}")
# epoch 0: ~0.4 ... epoch 4: ~0.01

Say the ritual out loud as you read the inner loop: zero the gradients, forward pass, compute the loss, backward pass, take a step. Every PyTorch training loop you will ever read, at any scale, is this same structure with different ingredients.

9. Evaluation: eval mode and no_grad

Measuring a trained model needs two switches: model.eval() puts layers like dropout into inference behaviour, and torch.no_grad() stops gradient tracking, saving memory and time. Continue from Example 8’s trained model.

# (run immediately after Example 8, reusing model and data)
model.eval()                          # inference behaviour for dropout etc.
with torch.no_grad():                 # no gradient bookkeeping needed
    logits = model(features)
    predicted = logits.argmax(dim=1)  # highest-scoring class per row
    accuracy = (predicted == labels).float().mean()
print(f"Accuracy: {accuracy:.2%}")    # ~100% on this easy synthetic data
model.train()                         # switch back before more training

The clouds are far apart, so near-perfect accuracy just proves the pipeline works. The habit to build is the pairing: eval()plus no_grad() for every evaluation, then train() before any further training. Forgetting these switches is a classic source of silently wrong results in models that use dropout or batch norm.

10. Saving, loading, and freezing

A trained model should outlive its script. state_dict saves the weights, load_state_dict restores them, and freezing parameters with requires_grad = False is the mechanism behind transfer learning.

import torch
import torch.nn as nn
model = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 2))
# save weights only (the recommended way)
torch.save(model.state_dict(), "classifier.pt")
# later, or in another script: rebuild the architecture, then load
restored = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 2))
restored.load_state_dict(torch.load("classifier.pt"))
restored.eval()
# transfer learning pattern: freeze early layers, retrain the head
for param in restored[0].parameters():
    param.requires_grad = False       # the first Linear stops learning
trainable = [p for p in restored.parameters() if p.requires_grad]
print(f"Trainable tensors remaining: {len(trainable)}")   # just the head

state_dict saves weights rather than the whole object, which is why you rebuild the architecture before loading, a deliberate design that keeps saved models portable. The freezing pattern is exactly how pretrained models are adapted: lock the early layers that learned general features, retrain the head on your task, and use a low learning rate such as 1e-5 for anything left unfrozen.

Work through these and you will have touched the whole article: tensors, layers, activations, losses, autograd, optimizers, data loading, the training loop, evaluation, and persistence. Two closing habits from the guide are worth pinning up. The learning rate is the hyperparameter that matters most, and searching it on a log scale beats a linear grid. And before any long training run, deliberately overfit a single batch: if your pipeline cannot drive the loss to near zero on 32 samples, something is wired wrong, and it is far cheaper to find out in thirty seconds than after an hour of training.

See you soon, Andrei.