A
| Term | Definition |
| AUC | See Area Under the Curve. |
| Accuracy | The share of predictions that are correct. Almost useless on imbalanced data, where predicting the majority class every time can score 99%. |
| Activation Function | The nonlinear function applied to a neuron’s weighted sum, such as ReLU, sigmoid or tanh. Without it a deep network collapses mathematically into a single linear layer. |
| Active Learning | Training a model iteratively, selecting the most informative unlabelled examples for a human to label next. Worth the machinery when labels are expensive and data is abundant. |
| AdaBoost | A boosting method that reweights misclassified examples so later weak learners focus on them. Effective and sensitive to noisy labels, since it keeps escalating attention on the examples it cannot get right. |
| Adam | An optimiser combining momentum with per-parameter adaptive learning rates. The default first choice for deep networks, though well-tuned SGD sometimes generalises better. |
| Adversarial Example | An input perturbed slightly, often imperceptibly, so a model misclassifies it. Evidence that high accuracy does not imply the model learned what you think. |
| Agent | A system that plans and takes actions in an environment or across tools to pursue a goal, rather than producing a single output. In LLM terms, a model in a loop with tools and memory. |
| Alignment | Making a model’s behaviour match human intent and values, through techniques such as instruction tuning and reinforcement learning from human feedback. |
| Anomaly Detection | Identifying observations that do not fit the learned pattern of normality. Usually unsupervised, because the interesting anomalies are precisely the ones absent from the training data. |
| Area Under the Curve | The area under the ROC curve, the probability the model ranks a random positive above a random negative. Threshold-free, and it flatters models on heavily imbalanced data, where precision-recall AUC is more informative. |
| Attention | The mechanism that lets a model weight all positions in a sequence when computing a representation for one of them, computed as softmax over scaled query-key dot products applied to values. The core operation of the transformer. |
| Autoencoder | A network trained to reconstruct its input through a narrow bottleneck, learning a compressed representation. Used for dimensionality reduction, denoising and anomaly detection. |
| AutoML | Automated search over preprocessing, models and hyperparameters. A strong baseline generator, and no substitute for framing the problem or checking for leakage. |
B
| Term | Definition |
| BERT | An encoder-only transformer pre-trained with masked language modelling, designed for understanding tasks rather than generation. Still a sensible choice for classification and retrieval where a generative model is overkill. |
| Backpropagation | The algorithm that computes gradients of the loss with respect to every weight by applying the chain rule backwards through the network. Training is gradient descent; backpropagation is how the gradients are obtained. |
| Bagging | Training models on bootstrap resamples and averaging them, reducing variance without increasing bias. Random forests are bagging plus feature subsampling. |
| Batch | The set of examples processed before one weight update. Larger batches give steadier gradients and less regularising noise, and they need a matched learning rate. |
| Batch Normalisation | Normalising layer activations across the batch during training, which stabilises and speeds up learning. Its behaviour differs between training and inference, a classic source of deployment bugs. |
| Bayesian Optimisation | Hyperparameter search that builds a probabilistic model of the objective and samples where improvement is most likely. More sample-efficient than grid or random search when each training run is expensive. |
| Bias | In a neuron, the constant term added to the weighted sum. In error decomposition, the part of error from a model too simple to capture the pattern. In fairness, systematic disadvantage to a group. Three unrelated meanings sharing one word. |
| Bias-Variance Tradeoff | The tension between models too simple to fit the signal and models so flexible they fit the noise. Regularisation, more data and ensembling are the standard ways to buy one without paying too much of the other. |
| Binary Classification | Prediction between two classes. Almost every subtlety in evaluation, from thresholds to imbalance, shows up here first. |
| Boosting | Building an ensemble sequentially, each model correcting the errors of those before it. Usually the strongest family on tabular data, and more prone to overfitting than bagging if left unregularised. |
C
| Term | Definition |
| CNN | See Convolutional Neural Network. |
| Calibration | The degree to which predicted probabilities match observed frequencies, so events predicted at 70% happen about 70% of the time. Ranking well and being calibrated are different things, and decisions that use probabilities need both. |
| Catastrophic Forgetting | A network losing previously learned capability while being trained on new data. The reason naive continual fine-tuning degrades a model on everything it used to do. |
| Class Imbalance | A large disparity in class frequencies. It breaks accuracy, distorts thresholds, and is handled by resampling, class weights or threshold tuning rather than by pretending it is not there. |
| Classification | Predicting a discrete label. |
| Clustering | Grouping unlabelled data by similarity. There is no ground truth to validate against, so the number of clusters and the notion of similarity are assumptions you impose, not findings. |
| Confusion Matrix | The table of true and false positives and negatives. Read it before any single-number metric, because it shows which kind of error the model actually makes. |
| Convergence | The point at which further training stops materially improving the objective. Convergence of the loss is not the same as a good model, only a settled one. |
| Convolutional Neural Network | A network using learned filters slid across the input to exploit local structure and spatial invariance. Dominant for images, and largely displaced by transformers at large scale. |
| Cost Function | See Loss Function. |
| Cross-Entropy | The standard loss for classification, penalising confident wrong predictions heavily. Its gradient behaviour is why it beats squared error for probabilistic outputs. |
| Cross-Validation | Rotating held-out folds to estimate performance more stably than a single split. Use grouped or time-based folds when observations are related, or the estimate will be optimistic. |
D
| Term | Definition |
| DBSCAN | A density-based clustering algorithm that finds arbitrarily shaped clusters and labels sparse points as noise. Needs no cluster count, and is sensitive to its density parameters and to unscaled features. |
| Data Augmentation | Expanding training data with label-preserving transformations such as crops, flips or paraphrases. Cheap regularisation, provided the transformations really do preserve the label. |
| Decision Boundary | The surface in feature space where the predicted class changes. Its shape is the clearest picture of what a model has actually learned. |
| Decision Tree | A model that splits data by feature thresholds into a tree of decisions. Interpretable and unstable: small data changes produce very different trees, which is exactly why ensembles of them work. |
| Deep Learning | Machine learning with many-layered neural networks that learn their own feature representations. Its advantage appears with large data and unstructured inputs, and rarely on small tabular problems. |
| Diffusion Model | A generative model trained to reverse a gradual noising process, generating samples by denoising from noise. The basis of current image and video generation. |
| Dimensionality Reduction | Projecting data into fewer dimensions while preserving structure, through PCA, UMAP, t-SNE and autoencoders. |
| Discriminative Model | A model of the boundary between classes, learning the conditional probability of the label given the input. Contrast generative models, which model how the data itself is produced. |
| Distillation | Training a small student model to reproduce a large teacher’s outputs. Retains much of the capability at a fraction of the serving cost. |
| Dropout | Randomly deactivating units during training so the network cannot rely on any single path. Applied at training time only, and forgetting to disable it at inference produces noisy predictions. |
E
| Term | Definition |
| Early Stopping | Halting training when validation performance stops improving. The simplest effective regulariser, and it needs a validation set kept separate from the test set. |
| Elastic Net | Regularisation combining L1 and L2 penalties, giving sparsity while handling correlated predictors better than LASSO alone. |
| Embedding | A learned dense vector representation in which semantic similarity corresponds to geometric closeness. Underpins retrieval, recommendation and everything built on vector search. |
| Ensemble | A combination of several models whose errors partly cancel. Reliably better than a single model, and harder to explain and to serve. |
| Epoch | One complete pass over the training data. |
| Error Analysis | Manually inspecting a sample of the model’s mistakes to find patterns. Consistently the highest-value hour in a modelling project, and consistently the one skipped. |
| Explainability | Techniques for describing why a model produced an output, including SHAP, LIME and attention inspection. An explanation is a model of the model, and can be wrong. |
| Exploding Gradient | Gradients growing uncontrollably through backpropagation, destabilising training. Controlled by gradient clipping and careful initialisation. |
F
| Term | Definition |
| F1 Score | The harmonic mean of precision and recall. A single number for imbalanced problems, and it hides which of the two is weak, so report the components too. |
| Feature Importance | A score for each input’s contribution to predictions. Impurity-based importances are biased toward high-cardinality features, so prefer permutation importance or SHAP. |
| Feature Selection | Choosing a subset of inputs to keep. Improves interpretability and often generalisation, and must happen inside cross-validation, since selecting on the full dataset leaks. |
| Few-Shot Learning | Getting useful behaviour from a handful of examples, in modern practice usually by placing examples in the prompt rather than by training. |
| Fine-Tuning | Continuing training of a pre-trained model on task-specific data. Cheaper than training from scratch, and it can degrade general capability if pushed too far. |
| Foundation Model | A large model pre-trained on broad data and adapted to many downstream tasks. The economic shift is that capability is now bought and adapted rather than built. |
G
| Term | Definition |
| GAN | Generative Adversarial Network, a generator and discriminator trained against each other. Historically important for image generation, largely superseded by diffusion models. |
| Generalisation | Performing well on data never seen in training. The only thing that matters, and the only thing training performance does not measure. |
| Generative Model | A model of how the data is produced, able to sample new examples from it. |
| Gini Impurity | A measure of class mixedness used to choose splits in decision trees. Nearly interchangeable with entropy in practice. |
| Gradient | The vector of partial derivatives of the loss with respect to parameters, pointing in the direction of steepest increase. Training moves against it. |
| Gradient Boosting | Boosting where each new model fits the residual errors of the ensemble so far. Its implementations, XGBoost, LightGBM and CatBoost, remain the strongest default for tabular data. |
| Gradient Descent | Iteratively adjusting parameters against the gradient to reduce loss. Every modern training procedure is a variation on it. |
| Grid Search | Exhaustively evaluating a fixed grid of hyperparameter combinations. Simple and wasteful, since most hyperparameters barely matter and random search finds the ones that do more quickly. |
| Ground Truth | The values treated as correct for training and evaluation. When they are themselves a proxy, the model learns the proxy and its flaws. |
H
| Term | Definition |
| Hallucination | A generated statement that is fluent and false. A consequence of models optimised to produce likely text rather than verified text, mitigated by retrieval grounding and by checking rather than eliminated. |
| Hidden Layer | Any layer between input and output. Depth buys the ability to compose features hierarchically. |
| Hierarchical Clustering | Building a tree of nested clusters by repeatedly merging or splitting. Gives a dendrogram rather than a fixed number of clusters, which is useful when the right number is unknown. |
| Hyperparameter | A setting chosen before training rather than learned from data, such as learning rate, depth or regularisation strength. Tune on validation data only. |
| Hyperparameter Tuning | Searching hyperparameter space for the best validation performance, by grid, random or Bayesian search. The gains are usually smaller than better features or more data. |
I
| Term | Definition |
| Inference | Using a trained model to produce predictions. Its cost, latency and reliability decide whether a model is deployable, and they are rarely considered during modelling. |
| Information Gain | The reduction in entropy achieved by a split, used to grow decision trees. |
| Isolation Forest | An anomaly detection method that isolates points with random splits, on the logic that anomalies need fewer splits to separate. Fast and effective in high dimensions. |
K
| Term | Definition |
| KL Divergence | A measure of how much one probability distribution differs from another. Asymmetric, and central to variational methods, distillation and the objectives used for model alignment. |
| K-Fold Cross-Validation | Splitting into k folds and training k times, each fold serving once as validation. |
| K-Means | Clustering that assigns points to the nearest of k centroids and updates the centroids until stable. Requires k in advance, assumes roughly spherical clusters of similar size, and is sensitive to scaling and initialisation. |
| K-Nearest Neighbours | Predicting from the labels of the closest training examples. No training cost and expensive inference, and it degrades badly in high dimensions where all distances converge. |
| Kernel | In an SVM, a function computing similarity in an implicit higher-dimensional space. In a CNN, the learned filter slid across the input. |
| Kernel Trick | Computing inner products in a high-dimensional feature space without ever constructing the coordinates, which is what makes nonlinear SVMs tractable. |
L
| Term | Definition |
| L1 Regularisation | A penalty on the absolute size of coefficients, which drives some to exactly zero and therefore performs feature selection. |
| L2 Regularisation | A penalty on squared coefficient size, shrinking all coefficients smoothly without eliminating any. Handles correlated predictors better than L1. |
| LASSO | Linear regression with an L1 penalty, producing sparse models. |
| LLM | Large Language Model, a transformer trained on large text corpora to predict the next token, then adapted through instruction tuning and preference optimisation. |
| LSTM | Long Short-Term Memory, a recurrent architecture with gates that carry information across long spans. Largely replaced by transformers, still reasonable for small sequential problems. |
| LoRA | Low-Rank Adaptation, fine-tuning by training small low-rank matrices alongside frozen original weights. Cuts the memory and storage cost of adaptation enormously and allows many task-specific adapters over one base model. |
| Latent Space | The learned internal representation space of a model, where positions correspond to meaningful structure rather than raw input values. |
| Learning Curve | Performance plotted against training set size or epochs. The fastest diagnostic for whether the fix is more data, a bigger model, or better features. |
| Learning Rate | The step size taken against the gradient. Too high and training diverges, too low and it crawls or gets stuck. The most consequential hyperparameter in deep learning. |
| Linear Regression | Fitting a linear relationship between inputs and a continuous output by least squares. Still the right first model, because it sets a baseline and its failures are informative. |
| Log Loss | See Cross-Entropy. |
| Logistic Regression | Linear modelling of the log-odds of a binary outcome, giving calibrated probabilities and readable coefficients. The baseline every classifier should be compared against. |
| Loss Function | The quantity training minimises. It encodes what you consider a mistake, so choosing it is a business decision disguised as a technical one. |
M
| Term | Definition |
| MAE | Mean Absolute Error, average absolute deviation. Less sensitive to outliers than RMSE and in the units of the target, so it communicates well. |
| MSE | Mean Squared Error, average squared deviation. Penalises large errors heavily and is not in the target’s units, which is why RMSE is usually reported instead. |
| Masking | Hiding parts of the input or of future positions so a model cannot use them. Causal masking is what makes a decoder predict the next token rather than copy it. |
| Mini-Batch | A subset of the data used for one gradient update, the practical middle ground between full-batch and single-example updates. |
| Model Drift | Degradation of a deployed model as the world moves away from its training distribution. Detected by monitoring inputs and outcomes, not by waiting for a complaint. |
| Momentum | Accumulating a running average of past gradients to smooth updates and escape shallow local structure. |
| Multi-Class Classification | Prediction across three or more mutually exclusive classes. |
| Multi-Label Classification | Prediction where each example may carry several labels at once, requiring per-label outputs rather than a softmax. |
N
| Term | Definition |
| Naive Bayes | A classifier applying Bayes’ theorem while assuming features are conditionally independent. The assumption is nearly always false and the classifier is often fine anyway, especially for text. |
| Neural Network | A model of layered weighted sums passed through nonlinearities, trained by gradient descent. |
| Normalisation | Rescaling inputs so features are comparable in magnitude. Essential for gradient-based and distance-based methods, irrelevant for trees, and it must be fitted on training data only. |
O
| Term | Definition |
| Objective Function | The quantity being optimised, minimised as a loss or maximised as a reward. |
| One-Hot Encoding | Representing a categorical value as a binary indicator per category. Explodes with high cardinality, which is where target or hashing encodings earn their place. |
| Online Learning | Updating a model continuously as data arrives rather than retraining in batches. Adapts fast and can be corrupted fast, so it needs guardrails. |
| Optimiser | The algorithm that turns gradients into parameter updates, such as SGD, Adam or AdamW. |
| Out-of-Bag Estimate | A validation estimate obtained in bagging from the examples each bootstrap sample left out. Free validation without a separate split. |
| Overfitting | Fitting noise as if it were signal. Visible as a widening gap between training and validation performance. |
| Oversampling | Increasing the representation of a minority class by duplicating or synthesising examples. It changes the base rate, so recalibrate probabilities afterwards. |
P
| Term | Definition |
| PCA | Principal Component Analysis, projection onto orthogonal directions of maximum variance. Linear, sensitive to scaling, and the components are usually uninterpretable, so it trades explanation for compactness. |
| Parameter | A value learned during training, such as a weight. Distinct from a hyperparameter, which is set beforehand. |
| Pooling | Downsampling a feature map by summarising regions, typically with a maximum or average, reducing size and adding translation tolerance. |
| Positional Encoding | Information added to token representations so a transformer knows sequence order, since attention itself is order-blind. Rotary encodings are the current standard and part of why long context works. |
| Precision | Of the cases predicted positive, the share that really are. The metric that matters when a false positive is expensive. |
| Precision-Recall Curve | Precision plotted against recall across thresholds. More informative than ROC on heavily imbalanced problems, where ROC looks deceptively good. |
| Pre-Training | The initial large-scale self-supervised training phase that produces general capability, before any task-specific adaptation. |
| Prompt | The input given to a language model, including instructions, context and examples. Its structure changes output quality enough that prompt design is a genuine engineering surface. |
| Pruning | Removing parts of a model that contribute little, whether branches of a tree or weights of a network, to reduce overfitting or serving cost. |
Q
| Term | Definition |
| Q-Learning | A reinforcement learning method that learns the expected return of taking an action in a state, without needing a model of the environment. |
| Quantisation | Storing and computing weights at lower numeric precision, such as 8-bit or 4-bit. Cuts memory and latency substantially for a small and usually acceptable quality loss. |
R
| Term | Definition |
| RAG | Retrieval Augmented Generation, retrieving relevant documents and placing them in the prompt so the model answers from supplied evidence. The practical fix for stale knowledge and unsupported claims, and its quality is bounded by the retrieval step. |
| RLHF | Reinforcement Learning from Human Feedback, training a reward model on human preference comparisons and optimising the policy against it. What turns a raw pre-trained model into a usable assistant. |
| RMSE | Root Mean Squared Error, the square root of MSE, expressed in the units of the target. Dominated by large errors, so check it alongside MAE. |
| ROC Curve | True positive rate against false positive rate across thresholds. Threshold-independent, and optimistic when negatives vastly outnumber positives. |
| Random Forest | An ensemble of decision trees trained on bootstrap samples with random feature subsets. Strong out of the box, hard to overfit, and less accurate than tuned gradient boosting on most tabular tasks. |
| Random Search | Sampling hyperparameter combinations at random. Beats grid search in practice because it explores more distinct values of the few parameters that actually matter. |
| Recall | Of the cases that really are positive, the share the model finds. The metric that matters when a miss is expensive. |
| Recurrent Neural Network | A network that processes sequences one step at a time, carrying a hidden state forward. Superseded by transformers, which parallelise instead of stepping. |
| Regression | Predicting a continuous value. |
| Regularisation | Any constraint discouraging complexity to improve generalisation, including L1, L2, dropout, early stopping and data augmentation. |
| Reinforcement Learning | Learning a policy through rewards received from acting in an environment. Powerful where the objective is a long-run outcome rather than a labelled example, and notoriously sensitive to reward specification. |
| ReLU | The rectified linear unit, outputting the input when positive and zero otherwise. Cheap, effective, and it can leave units permanently dead, which is what variants such as leaky ReLU address. |
| Residual Connection | A shortcut adding a layer’s input to its output. What makes very deep networks trainable, by giving gradients an unobstructed path backwards. |
| Ridge Regression | Linear regression with an L2 penalty. |
S
| Term | Definition |
| SGD | Stochastic Gradient Descent, updating parameters from small random batches rather than the full dataset. The noise in the updates is itself mildly regularising. |
| SHAP | SHapley Additive exPlanations, attributing a prediction to its features using a game-theoretic allocation. The most principled common explanation method, and expensive on large models. |
| SMOTE | Synthetic Minority Oversampling Technique, creating synthetic minority examples by interpolating between neighbours. Helps sometimes, and it can manufacture examples in regions where the minority class does not actually live. |
| Scaling Laws | The empirical relationships showing model loss falling predictably with more parameters, data and compute. They are why capability increases were forecastable rather than accidental. |
| Self-Attention | Attention where queries, keys and values all come from the same sequence, letting each position attend to every other. |
| Self-Supervised Learning | Learning from labels derived automatically from the data itself, such as predicting the next or a masked token. It unlocked training on unlabelled text at scale. |
| Semi-Supervised Learning | Learning from a small labelled set plus a large unlabelled one. |
| Sigmoid | A function squashing any real number into zero to one, used for binary outputs and gates. Saturates at the extremes, which stalls gradients. |
| Silhouette Score | A measure of how well each point sits in its cluster relative to the next nearest. One of the few usable ways to choose a cluster count. |
| Softmax | A function turning a vector of scores into a probability distribution. Used for multi-class outputs and inside attention. |
| Supervised Learning | Learning a mapping from inputs to known labels. |
| Support Vector Machine | A classifier finding the maximum-margin separating boundary, extended to nonlinear boundaries by kernels. Strong on small and medium datasets, and it scales poorly. |
| Synthetic Data | Artificially generated data used to augment or replace real data. Useful for rare cases and privacy, and it can only contain the structure that was put into it. |
T
| Term | Definition |
| Target Leakage | Information unavailable at prediction time reaching the training features. Produces excellent validation scores and worthless production performance, and it is the single most common serious modelling bug. |
| Temperature | A parameter scaling the logits before sampling, controlling randomness in generation. Low values give repetitive determinism, high values give incoherence. |
| Tensor | A multi-dimensional array, the data structure of deep learning frameworks. |
| Token | The unit a language model reads and generates, typically a subword fragment. Cost, context limits and latency are all measured in tokens, not words. |
| Tokenisation | Splitting text into tokens through a learned subword vocabulary such as byte-pair encoding. It determines how efficiently a language is represented, which is why some languages cost several times more per sentence. |
| Transfer Learning | Reusing a model trained on one task as the starting point for another. The default approach whenever the target dataset is small. |
| Transformer | The architecture built on self-attention and feed-forward blocks with residual connections and normalisation. It replaced recurrence with parallelism, which is what made training at current scale possible. |
| t-SNE | A nonlinear method for visualising high-dimensional data in two dimensions, preserving local neighbourhoods. Cluster sizes and between-cluster distances in the plot are not meaningful, which people over-read constantly. |
U
| Term | Definition |
| UMAP | A nonlinear dimensionality reduction method, faster than t-SNE and better at preserving global structure. Still a visualisation aid rather than evidence. |
| Underfitting | A model too simple or too constrained to capture the pattern, showing poor performance on training data as well as validation. |
| Undersampling | Reducing the majority class to balance classes. Cheap, and it throws away real data. |
| Unsupervised Learning | Learning structure from unlabelled data, through clustering, dimensionality reduction or density estimation. |
V
| Term | Definition |
| Validation Set | Data used for tuning and model selection, distinct from training and test data. |
| Vanishing Gradient | Gradients shrinking toward zero as they propagate backwards, stalling learning in early layers. Addressed by ReLU activations, residual connections and normalisation. |
| Variance | In error decomposition, the part of error caused by sensitivity to the particular training sample. |
| Vector Database | A store optimised for approximate nearest neighbour search over embeddings. The retrieval half of retrieval augmented generation. |
W
| Term | Definition |
| Weak Learner | A model only slightly better than chance, the building block boosting combines into a strong one. |
| Weight | A learned coefficient on a connection or feature. |
| Weight Decay | L2 regularisation applied in the optimiser, shrinking weights toward zero at each step. |
| Word Embedding | A dense vector representation of a word learned from context, as in word2vec or GloVe. Superseded by contextual embeddings, which give a word a different vector in each sentence. |
X
| Term | Definition |
| XAI | Explainable Artificial Intelligence, the field of making model behaviour understandable to humans. |
| XGBoost | A widely used gradient boosting implementation with strong regularisation and efficient handling of sparse data. Together with LightGBM and CatBoost, still the benchmark to beat on tabular problems. |
Z
| Term | Definition |
| Zero-Shot Learning | Performing a task with no task-specific examples, relying on instructions and pre-trained knowledge alone. |
