A
| Term | Definition |
| A/A Test | An experiment where both arms receive the identical experience. Run to validate the testing setup before trusting it: correct randomisation, no sample ratio mismatch, no tracking bias, and a false positive rate matching the nominal alpha. A statistically significant A/A result is a signal that the infrastructure is broken, not that A beats A. |
| A/B Test | A controlled online experiment comparing two experiences, a control (A) and a treatment (B), by randomly assigning users to each and comparing a chosen metric with a statistical test. Randomisation is what makes the measured difference causal rather than correlational. |
| A/B Testing | The ongoing practice of using A/B tests as the decision method for product and marketing changes, including the surrounding process: hypothesis, design, sample size planning, analysis, and rollout decision. |
| A/B/n Test | An A/B test with more than one treatment: one control plus n-1 variants running at the same time. Traffic per arm drops and the risk of a false positive rises across the family of comparisons, so it needs a multiple comparisons adjustment or a designated primary variant. |
| AGILE A/B Test | A test run under the AGILE statistical method, an adaptive sequential design that combines alpha-spending with futility stopping so results can be monitored throughout and stopped early for a win or for hopelessness, while keeping the overall error rate at the planned level. Typically reaches a decision on far less traffic than a fixed-sample test of the same power. |
| AOV | See Average Order Value. |
| ARPU | See Average Revenue Per User. |
| ASD | See Adaptive Sequential Design. |
| Absolute Change | The difference between treatment and control expressed in the metric’s own units, for example a conversion rate moving from 4.0% to 4.8% is an absolute change of 0.8 percentage points. Always state whether a reported change is absolute or relative, since the same result reads as either 0.8 or 20%. |
| Absolute Difference | The arithmetic difference between two measured values, treatment minus control, in original units. Same quantity as absolute change, used when describing the gap rather than the movement. |
| Actual Significance | The true type I error rate of a testing procedure as it was actually run, as opposed to the nominal level written on the report. Repeatedly checking a fixed-sample test and stopping at the first significant reading can push a nominal 5% up past 20% or 30%, and that inflated number is the actual significance. |
| Actual p-value | The p-value corrected for how the test was really conducted, accounting for interim looks, early stopping rules, and multiple variants. It is the number that supports a decision. The naive p-value from a fixed-sample formula is only equal to it when the test was genuinely analysed once, at the planned sample size. |
| Adaptive Sequential Design | A design that plans interim analyses in advance and permits pre-specified adaptations at them: stopping early for efficacy, stopping early for futility, or re-estimating the sample size. Because the adaptations are planned and the error is budgeted across looks, error control is preserved, unlike ad hoc peeking. |
| Alpha-Spending | The technique of splitting the total type I error budget (alpha) across several interim analyses so that the error spent at each look sums to the overall alpha. This is what makes it legitimate to monitor a test in flight instead of waiting for a single final read. |
| Alpha-spending Function | The rule that decides how much of the alpha budget is released at each interim look, as a function of the information fraction. Conservative shapes (O’Brien-Fleming style) spend almost nothing early and keep power near the end, while flatter shapes (Pocock style) spend evenly and stop earlier at the cost of a larger maximum sample size. Lan-DeMets is the framework that lets looks happen at unplanned times. |
| Alternative Hypothesis | The hypothesis that a real effect exists, complementary to the null. In a one-sided online test it is usually “the treatment is better than the control by at least the minimum effect of interest”. Its direction and size determine the power calculation. |
| Argument from Coincidence | The reasoning behind a significance test: if the null hypothesis were true, data as extreme as what was observed would be an implausible coincidence, so the null is rejected. It explains why a p-value is evidence against the null rather than a probability that the null is true. |
| Average | The arithmetic mean, the sum of the values divided by their count. Sensitive to outliers, which matters for revenue metrics where a handful of large orders can move the mean of an arm on their own. |
| Average Order Value | Total revenue divided by the number of orders. Measures basket size, not customer value, and it moves when the mix of buyers changes, so a variant that suppresses small orders can lift AOV while lowering total revenue. |
| Average Revenue Per User | Total revenue divided by the number of users exposed to the arm, not the number of buyers. Combines conversion rate and order value into one number, which makes it the usual primary metric for revenue tests but also a high variance one that needs more traffic. |
| Average Sample Size | The expected number of users a sequential test consumes before it stops, averaged over its stopping distribution. It is the efficiency measure that matters when comparing a sequential design to a fixed-sample one, since the sequential test’s maximum sample size is larger but its average is smaller. |
B
| Term | Definition |
| Bandit | See Multi-Armed Bandit. |
| Baseline | The current experience and the metric level it produces, the reference every variant is judged against. A personalisation programme needs a permanent holdout to keep a baseline visible once targeting is switched on, otherwise there is nothing left to compare to. |
| Baseline Conversion Rate | The control’s conversion rate, and the main input to sample size planning. Required traffic is very sensitive to it, so estimate it from recent comparable traffic rather than an annual average that smooths over seasonality. |
| Bayes Factor | The ratio of how likely the observed data is under one hypothesis versus the other, a Bayesian measure of relative evidence. Unlike a p-value it can express support for the null, not merely a failure to reject it. |
| Bayesian A/B Testing | A framework that places a prior on the effect and reports a posterior, producing statements such as “94% probability that B beats A” and “expected loss if we ship B”. It is not automatically immune to peeking: monitoring continuously still inflates the rate of wrong ship decisions unless the decision rule accounts for it. |
| Behavioural Targeting | Personalising on what a visitor has done, such as pages viewed, products browsed, or past purchases, rather than on who they are demographically. |
| Beta Distribution | A distribution over values between 0 and 1, the conjugate prior for the binomial. It is the standard prior for a conversion rate in Bayesian testing because the posterior stays Beta, updated simply by adding conversions and non-conversions. |
| Between-Subjects Design | Each user sees exactly one variant, the standard design for online tests. Contrast with within-subjects, where the same user sees both and order effects contaminate the comparison. |
| Bias | Systematic error that pushes an estimate away from the truth in a consistent direction, unlike random noise which averages out with more data. Common sources are broken randomisation, tracking that fails more on one variant, and defining exposure after the point where users drop out. |
| Binary Metric | A metric that is yes or no per unit, such as converted or clicked. Analysed as a proportion, and its variance is fixed by the rate itself, which is why very low conversion rates need so much traffic. |
| Binomial Distribution | The distribution of the number of successes in a fixed number of independent trials with constant success probability. The model underlying every conversion rate test. |
| Blocking | Grouping similar units before randomisation and randomising within each group, so known sources of variation such as device, country, or new versus returning are balanced across arms. Reduces variance and therefore the traffic needed. |
| Bonferroni Correction | The simplest multiple comparisons correction: divide alpha by the number of comparisons. Always valid but conservative, so with several variants it costs real power. Holm or Benjamini-Hochberg give the same protection with more power. |
| Bounce Rate | The share of sessions that end after a single interaction. A weak primary metric because it shifts with tracking implementation and says nothing about value delivered. |
| Bucketing | Assigning a user to a variant, normally by hashing a stable identifier together with the experiment name so the assignment is deterministic and survives across visits without needing to be stored. |
C
| Term | Definition |
| CDP | See Customer Data Platform. |
| CLV | See Customer Lifetime Value. |
| CRO | See Conversion Rate Optimisation. |
| CTA | See Call to Action. |
| CTR | See Click-Through Rate. |
| CUPED | Controlled Experiment Using Pre-Experiment Data. A variance reduction method that adjusts each user’s metric using their behaviour before the test started, often cutting required sample size by 20% to 50% without changing what is being estimated. Only helps for users who have pre-period history. |
| Call to Action | The element that asks the user to act, such as a button or link. A frequent test subject, and one where the metric must be downstream value rather than clicks on the CTA itself. |
| Carryover Effect | When exposure in one experiment or period changes behaviour in a later one, contaminating the later measurement. The reason to use washout periods and to re-randomise buckets between sequential tests on the same surface. |
| Central Limit Theorem | The result that the distribution of sample means approaches normal as sample size grows, whatever the shape of the underlying data. It justifies z and t tests on web data, but revenue is skewed enough that “large enough” can mean tens of thousands of users per arm. |
| Change Aversion | A temporary metric drop caused by existing users being disrupted by an unfamiliar interface rather than by the new design being worse. Diagnose it by looking at new users separately and by running past the adjustment period. |
| Chi-Squared Test | A test of association in a contingency table, sometimes used to compare conversions across variants. It gives no direction and no interval, so a z test on the difference in proportions is almost always more useful. |
| Click-Through Rate | Clicks divided by impressions. A proxy metric that is easy to move, including in ways that do not survive to revenue. |
| Client-Side Testing | Variants rendered by JavaScript in the browser after the original page loads. Quick to deploy without engineering help, but it introduces flicker and page weight, both of which bias results against the variant. |
| Cohort | A group of users sharing a defining event in the same time window, for example everyone acquired in March. |
| Cohort Analysis | Following cohorts over time to separate lifecycle effects from calendar effects. Complements experiments for long-run questions that a two-week test cannot answer. |
| Collaborative Filtering | A recommendation approach that uses patterns across users, the “people who bought this also bought” logic. Strong where interaction data is dense, weak on cold start for new users or new items. |
| Confidence Interval | The range of effect sizes compatible with the data at a stated confidence level. The guarantee is about the procedure: intervals built this way contain the true effect 95% of the time. More informative than a p-value because it shows both magnitude and precision. |
| Confidence Level | The long-run coverage of the interval procedure, one minus alpha. 95% is a convention, not a rule, and the appropriate level depends on the cost of being wrong. |
| Confounding | When some third factor is associated with both the variant a user gets and the outcome, corrupting the comparison. Randomisation removes it in expectation, which is why broken randomisation is fatal rather than untidy. |
| Content-Based Filtering | Recommending items by matching item attributes to a user’s revealed preferences, without reference to other users. Handles new items better than collaborative filtering. |
| Continuous Metric | A metric that takes any numeric value per unit, such as revenue or time on site. Higher variance than a binary metric, so it needs more traffic and usually some capping or trimming of extremes. |
| Control Group | The arm receiving the unchanged experience. In personalisation this becomes a permanent holdout, deliberately excluded from all targeting so the cumulative value of the programme stays measurable. |
| Conversion | Completion of the action the experiment is about. Define it precisely once, before launch, because the word silently means different things in different reports. |
| Conversion Rate | Converting units divided by exposed units. The denominator choice, users or sessions or visitors, changes both the number and its variance, so always state which is in use. |
| Conversion Rate Optimisation | The discipline of raising conversion through research, hypothesis, testing and iteration. Testing is one step inside it, not the whole of it. |
| Cookie | A browser-stored identifier used to keep variant assignment stable across visits. Deletion, tracking prevention, and multi-device use all cause assignment churn, inflating counted users relative to real people. |
| Counterfactual | What would have happened to the same users had they received the other variant. Unobservable for any individual, which is precisely what randomisation lets you estimate on average. |
| Coverage | The proportion of intervals produced by a procedure that actually contain the true value. Nominal 95% coverage degrades under peeking, or under normal approximations applied to very small counts. |
| Cross-Device Tracking | Linking one person’s activity across phone, tablet and desktop into a single identity. Without it, one person can be counted in two arms, which dilutes the measured effect toward zero. |
| Customer Data Platform | A system that unifies customer data from many sources into persistent profiles and pushes segments out to activation channels. The usual backbone for personalisation targeting. |
| Customer Lifetime Value | The expected total profit from a customer over the whole relationship. The metric most decisions should optimise, and the hardest to observe inside a two-week test, so experiments use short-run proxies that have been validated against it. |
D
| Term | Definition |
| Decision Rule | The pre-registered statement of which result leads to which action, written before launch. Without one, results get reinterpreted after the fact to justify whatever the team already wanted. |
| Degrees of Freedom | The number of independent pieces of information available for estimating variability. It shapes the t distribution, but at online sample sizes t and z agree to the decimal, so it rarely changes a decision. |
| Delta Method | A technique for deriving the variance of a ratio or transformed statistic from the variances of its components. How you get a correct interval for ratio metrics such as revenue per session, where the denominator is itself random. |
| Dependent Variable | The outcome being measured, in other words the metric. The variant assignment is the independent variable. |
| Deterministic Matching | Identity resolution using hard identifiers such as a login or hashed email. More accurate than probabilistic matching, but it only covers authenticated traffic. |
| Difference in Means | The estimator used for continuous metrics, treatment mean minus control mean. Its standard error drives the entire test. |
| Directional Hypothesis | A hypothesis that states which way the effect should go, tested one-sided. It buys power at the same traffic and is honest for most product changes, provided the direction is fixed before anyone looks at the data. |
| Dwell Time | Time spent with a page or piece of content before moving on. Ambiguous alone, since long dwell can mean engaged or can mean confused. |
| Dynamic Content | Page content assembled per user or segment at request time. The delivery mechanism behind most personalisation. |
E
| Term | Definition |
| Effect Size | The magnitude of the difference between arms, stated in absolute or relative terms. Significance says an effect probably exists, effect size says whether it is worth shipping. |
| Efficacy Boundary | In a sequential design, the threshold the test statistic must cross at an interim look for the test to stop early and declare a winner. |
| Engagement Rate | The share of exposed users who perform some defined interaction. Reasonable as a secondary or guardrail metric, dangerous as a primary one because it can be inflated without creating value. |
| Event | A recorded user action, the raw material for both experiment metrics and personalisation triggers. |
| Exclusion Criteria | The rules that decide which users are left out of analysis, such as bots, internal traffic, or users who never reached the tested page. They must be defined before launch and applied identically to every arm. |
| Expected Loss | In Bayesian testing, the average amount of metric you would give up by shipping a particular variant, averaged over the posterior. The usual Bayesian stopping rule is to ship once expected loss drops below a threshold small enough to be treated as negligible. |
| Experiment | A study in which the intervention is assigned by the experimenter rather than chosen by the subject. That assignment is what licenses a causal claim. |
| Experimental Unit | The thing being randomised, usually a user, sometimes a session, sometimes a whole market. Analysis has to happen at the same unit as randomisation or the standard errors are wrong. |
| Exploration versus Exploitation | The trade-off between gathering more information about uncertain options and cashing in on the best option known so far. Bandits manage it continuously; a classical test explores fully, then exploits. |
| Exposure | The moment a user actually encounters the tested difference. Counting from page entry rather than from exposure dilutes the effect with users who never saw the change. |
| External Validity | Whether a result generalises beyond the traffic, period and context it was measured in. A win on Black Friday traffic is not a general win. |
F
| Term | Definition |
| FDR | See False Discovery Rate. |
| FWER | See Family-Wise Error Rate. |
| Factorial Design | A design that varies several factors at once across all combinations, estimating main effects and interactions from the same traffic. Efficient for main effects, but interactions need far more traffic than people expect. |
| False Discovery Rate | The expected proportion of false positives among the results declared winners. The right error to control when screening many variants or many metrics, since it trades a tolerable share of false wins for much more power. |
| False Negative | Failing to detect an effect that is real, a type II error. The more common failure in underpowered programmes, and an invisible one, because the test simply reports no difference. |
| False Positive | Declaring an effect that is not real, a type I error. Every shipped false positive carries maintenance cost indefinitely, which is why alpha should reflect the cost of reverting. |
| Family-Wise Error Rate | The probability of at least one false positive across a family of comparisons. It climbs fast: five independent comparisons at 5% each give roughly a 23% chance of at least one false win. |
| Feature Flag | A runtime switch that turns functionality on or off per user. The delivery mechanism for server-side experiments and for staged rollouts. |
| Fixed-Sample Test | A test analysed once, at a sample size fixed in advance. Statistically the simplest option and the least traffic-efficient, and the design that peeking invalidates. |
| Flicker Effect | The brief flash of original content before a client-side variant renders over it. It depresses the variant’s numbers and is the main argument for server-side testing or an anti-flicker snippet. |
| Frequentist Statistics | The framework that treats parameters as fixed and probability as long-run frequency, producing p-values and confidence intervals. The default in most testing tools. |
| Funnel | The ordered sequence of steps toward a goal. Tests should report the whole funnel, because a variant that lifts step one and damages step three is a loss. |
| Funnel Analysis | Measuring drop-off at each step to find where users are lost. The usual source of test hypotheses. |
| Futility Stopping | Ending a sequential test early because the result can no longer plausibly reach significance. Saves the traffic that would otherwise be spent running a dead test to its planned end. |
G
| Term | Definition |
| Geo Experiment | Randomising whole geographies rather than individual users, used when the change cannot be split at user level, such as television spend, delivery pricing, or out-of-home. Very few units means very little power, so it needs matched markets and long run times. |
| Goal | The metric a test is built to move, and in most tools the name for the tracked conversion event. Keep the single primary goal separate from secondary and guardrail metrics, or every test finds something to celebrate. |
| Group Sequential Design | A design with a fixed number of pre-planned interim analyses and boundaries calculated so the overall error still lands at alpha. The classical form of sequential testing and the ancestor of alpha-spending approaches. |
| Guardrail Metric | A metric you have no intention of improving but refuse to damage, such as page load time, refund rate, or unsubscribe rate. Checked on every test to catch wins that are really just harm moved somewhere less visible. |
H
| Term | Definition |
| Hawthorne Effect | Behaviour changing because people know they are being observed. Rare in anonymous web testing, but real in usability sessions and internal betas, where it inflates the apparent success of a new design. |
| Heatmap | A visualisation aggregating clicks, cursor movement or scroll depth on a page. Good for generating hypotheses, useless for validating them, since there is no control to compare against. |
| Heterogeneous Treatment Effect | When the effect differs across segments, for example positive on mobile and negative on desktop. This is the entire justification for personalisation: if effects were uniform there would be nothing to personalise on. |
| Hierarchical Model | A model that pools information across segments or across past tests, pulling noisy segment estimates toward the overall effect. The principled way to read segment breakdowns without chasing noise. |
| Holdout Group | Users deliberately withheld from a treatment, or from an entire programme, so its cumulative effect stays measurable. Standard in personalisation and recommendations, where otherwise no baseline survives. |
| Holm-Bonferroni | A stepwise correction that compares ordered p-values against progressively looser thresholds. It gives the same family-wise guarantee as Bonferroni with strictly more power, so there is little reason to use plain Bonferroni instead. |
| Hypothesis | A falsifiable statement joining a change, an expected effect and a reason, for example “removing the coupon field will raise checkout completion because it sends users off to hunt for codes”. A change with no stated reason produces no learning whichever way it goes. |
| Hypothesis Testing | The formal procedure of stating a null, fixing an alternative and an alpha, computing a statistic, and deciding. It is a decision rule under uncertainty, not a measurement of truth. |
I
| Term | Definition |
| Identity Resolution | Deciding which observed identifiers belong to the same person. It sets the quality ceiling for both experiment assignment and personalisation accuracy. |
| Implicit Feedback | Preference inferred from behaviour such as clicks, dwell and purchases, rather than from stated ratings. Plentiful but noisy, and biased by whatever the system chose to show in the first place. |
| Impression | One instance of a piece of content being displayed. The denominator for click-through rate and the exposure record behind many personalisation tests. |
| Incrementality | The additional outcome genuinely caused by an intervention, over and above what would have happened anyway. The central question in both testing and marketing measurement, and the reason last-click attribution overstates almost everything. |
| Inconclusive Test | A test that ends without crossing its threshold. Not the same as evidence of no difference: it usually means the test never had the power to resolve an effect of the size that would have mattered. |
| Independent Variable | The thing being manipulated, which in an online test is the variant assignment. |
| Inference | Drawing conclusions about a population from a sample, with an explicit account of the uncertainty involved. |
| Information Fraction | The share of the planned total information, roughly the planned sample size, that has been collected by a given interim look. Alpha-spending functions are defined in terms of it. |
| Instrumentation | The tracking code and data collection that produce the experiment’s metrics. Most disputed results turn out to be instrumentation failures rather than statistical ones. |
| Intention to Treat | Analysing every user in the arm they were assigned to, whether or not they actually experienced the treatment. Conservative but unbiased, unlike per-protocol analysis, which drops non-compliers and quietly reintroduces selection bias. |
| Interaction Effect | When the effect of one factor depends on the level of another. Detecting one reliably takes roughly four times the sample size needed for the corresponding main effect. |
| Interim Analysis | A look at the data before the test has finished. Legitimate when planned and paid for with alpha-spending, and nothing but peeking when it is not. |
| Interleaving | An evaluation method for ranking that blends the outputs of two rankers into a single list per user and observes which source earns the clicks. Far more sensitive than a conventional A/B test for search and recommendation ranking. |
| Internal Validity | Whether the difference measured really was caused by the treatment within this experiment. Randomisation, sound instrumentation and no cross-arm contamination are what buy it. |
J
| Term | Definition |
| Journey | The full sequence of interactions a customer has with a business across channels and over time. The unit personalisation increasingly optimises, rather than the single page. |
| Journey Orchestration | Coordinating messages and experiences across channels and time according to where a customer sits in their journey, instead of optimising each touchpoint in isolation. |
K
| Term | Definition |
| KPI | See Key Performance Indicator. |
| Key Performance Indicator | A metric a team is accountable for. Not automatically a good experiment metric, since KPIs are often too slow or too noisy to move detectably within the life of a test. |
| Kill Switch | A mechanism for turning a variant or feature off immediately without a deployment. Worth having before launching anything risky, because the cost of a bad variant is measured in how long it stays live. |
| Kruskal-Wallis Test | A rank-based nonparametric test for differences across three or more groups, the analogue of one-way ANOVA. Sometimes reached for with skewed metrics, but it tests stochastic dominance rather than means, which is rarely the business question. |
L
| Term | Definition |
| Lagging Indicator | An outcome only visible well after the change, such as retention, churn or lifetime value. These are the real measures of success and far too slow to steer a two-week test. |
| Latency | The delay between request and response. A guardrail on nearly every test, since both personalisation and client-side testing add it, and added latency reliably costs conversion. |
| Leading Indicator | A fast-moving early proxy believed to predict a lagging outcome. Only trustworthy once the link to the lagging metric has actually been validated against past experiments. |
| Lift | The relative change between variant and control, the effect divided by the baseline. The default reporting format, and the reason absolute and relative results must always be labelled. |
| Local Maximum | The state where iterative small tests stop producing gains because the design is a well-polished version of the wrong concept. The argument for running occasional radical redesign tests alongside incremental ones. |
| Log Transformation | Taking logarithms of a skewed metric such as revenue to stabilise variance. It changes what is being estimated to a multiplicative effect on the geometric mean, which is not the effect on total revenue, so business decisions usually need the untransformed scale. |
| Long-Term Holdout | A holdout maintained for months rather than for the length of one test, used to check whether short-run wins persist. It frequently reveals that a year of stacked wins does not add up to the sum of its parts. |
| Lookalike Audience | An audience built by finding users who resemble a seed set of valuable customers. Useful for prospecting, and it inherits every bias present in the seed. |
M
| Term | Definition |
| MAB | See Multi-Armed Bandit. |
| MDE | See Minimum Detectable Effect. |
| MVT | See Multivariate Test. |
| Mann-Whitney U Test | A rank-based nonparametric test comparing two samples. Robust to skew, but it asks whether one sample tends to exceed the other rather than whether the means differ, so it can crown a winner that makes less money. |
| Marketing Mix Modelling | Regression on aggregate historical spend and outcomes to attribute results across channels. Fills the gaps where user-level experiments are impossible, at the cost of leaning on correlation, so calibrate it against geo experiments wherever you can run them. |
| Matched Market | A market selected to mirror a test market on size, seasonality and trend, used as the control in a geo experiment. The quality of the pre-period match is essentially the whole design. |
| Maximum Sample Size | The cap a sequential design plans for if no boundary is crossed earlier. It is larger than the fixed-sample equivalent, and that is the price paid for the option to stop early. |
| Mean | See Average. |
| Median | The middle value when observations are ordered. More robust to outliers than the mean, but businesses bank totals rather than medians, so a lift in the median is rarely the number to decide on. |
| Metric | A defined, computable measure of behaviour or business outcome. A metric needs its numerator, denominator and time window written down, or two reports will disagree about it forever. |
| Minimum Detectable Effect | The smallest effect the test is designed to detect at the chosen power and alpha. Set it from what would actually be worth shipping, not from whatever makes the sample size calculator return a comfortable number. |
| Monte Carlo Simulation | Repeatedly simulating an experiment under a known truth to measure a procedure’s real error rate and power. The most reliable way to see what peeking, or any novel stopping rule, genuinely costs. |
| Multi-Armed Bandit | Adaptive allocation that moves traffic toward better-performing arms while the test runs, trading clean measurement for reduced regret. Good for short-lived choices such as headlines, poor when you need a trustworthy effect estimate or when the best arm changes over time. |
| Multiple Comparisons Problem | The inflation of false positives caused by testing many variants, metrics or segments at the same alpha. Handled by Bonferroni, Holm or false discovery rate control, or avoided by committing to one primary metric. |
| Multivariate Test | A test that varies several page elements at once to estimate each element’s contribution and their interactions. Traffic-hungry, and in practice usually beaten by a sequence of focused A/B tests. |
N
| Term | Definition |
| NBA | See Next Best Action. |
| Next Best Action | A personalisation approach that scores the available actions for a customer at a given moment and takes the highest-value one, instead of following fixed campaign rules. |
| Nominal Significance Level | The alpha written into the plan, the false positive rate the procedure would deliver if it were followed exactly. It parts company with actual significance the moment anyone peeks. |
| Non-Inferiority Test | A test designed to show the variant is not worse than control by more than a stated margin. The right frame for migrations, replatforms and refactors, where the aim is to change the machinery without losing anything. |
| Nonparametric Test | A test that avoids assuming a particular distribution, working from ranks or resampling instead. Useful for badly skewed data, at the cost of answering a different question than the one about means. |
| Normal Distribution | The symmetric bell-shaped distribution that sample means converge toward. Web metrics themselves are almost never normal, and they do not need to be, since only the sampling distribution of the estimate has to be. |
| Novelty Effect | A short-lived lift caused by users reacting to something new rather than something better. Detect it by separating new from returning users and by checking whether the effect decays over the life of the test. |
| Null Hypothesis | The default position that there is no difference between arms. A test never proves it, it either accumulates enough evidence against it or fails to. |
O
| Term | Definition |
| OEC | See Overall Evaluation Criterion. |
| O’Brien-Fleming Boundary | A sequential boundary that is very strict at early looks and loosens toward the end. Retains nearly all the power of a fixed-sample test while still permitting early stopping when the effect is large. |
| Observational Study | Analysis of data where the treatment was not assigned by the analyst. Cheap and always available, and always exposed to confounding, which is why it generates hypotheses rather than settling them. |
| Omnichannel | Treating every channel as one continuous experience with shared state, rather than as separate silos with separate rules. |
| One-Sided Test | A test that places the entire alpha in one direction. More powerful at the same traffic when the direction is genuinely pre-committed, and indefensible when the direction is chosen after seeing which way the data went. |
| One-Tailed Test | See One-Sided Test. |
| Optimisation | Iteratively improving an experience against a defined objective. Distinct from testing, which is the measurement step inside it. |
| Outlier | An observation far from the rest, routine in revenue data where a single order can dominate an arm. Handle it with a capping or trimming rule declared before launch, never by removing the points that spoil the result. |
| Overall Evaluation Criterion | A single composite metric that combines the outcomes a test has to trade off, used to stop teams cherry-picking among many metrics. Hard to build honestly, and worth it because it forces the trade-offs to be agreed before launch. |
P
| Term | Definition |
| P-Value | The probability of observing data at least as extreme as what was seen, assuming the null hypothesis is true. It is not the probability that the null is true, and not the probability the result will replicate. |
| Parametric Test | A test that assumes a distributional form for the data or the estimator, such as t and z tests. At web sample sizes those assumptions are usually harmless when the target is a mean. |
| Peeking | Repeatedly checking a fixed-sample test and acting on the first significant reading. It can drive the real false positive rate past 30%, and it is the single most common reason online results fail to replicate. |
| Per-Protocol Analysis | Analysing only the users who actually received the treatment as intended. More intuitive than intention to treat and biased, because compliance is not random. |
| Personalisation | Adapting an experience to a user or segment using known or inferred attributes. Its value has to be proven against a holdout, since a programme cannot be evaluated by adding up the reported wins of its individual rules. |
| Pocock Boundary | A sequential boundary that spends alpha evenly across looks. Stops earlier than O’Brien-Fleming on moderate effects, at the cost of a larger maximum sample size and less power at the end. |
| Point Estimate | The single best-guess value for the effect, such as the observed difference in means. Always report it with an interval, because on its own it carries no information about precision. |
| Population | The full set of users the conclusion is meant to apply to. Fixing it in advance is what makes the sampling question answerable. |
| Post-Hoc Analysis | Analysis decided after seeing the data, segment digging above all. Legitimate for generating hypotheses, never for claiming a result, because the effective number of comparisons is unknown and large. |
| Posterior Distribution | In Bayesian analysis, the distribution of the effect after prior and data are combined. Everything a Bayesian test reports, from probability to beat control to expected loss, is a summary of it. |
| Power | The probability a test detects an effect of a given size when that effect is real, one minus the false negative rate. The usual 80% target means one real win in five is missed. |
| Power Analysis | The calculation linking baseline rate, minimum detectable effect, alpha and power to the required sample size. Run before launch it sets the run length, run after a null result it explains what the test could never have seen. |
| Practical Significance | Whether an effect is large enough to be worth acting on. With enough traffic a trivial effect becomes statistically significant, which is why the ship threshold should be an effect size and not a p-value. |
| Pre-Registration | Writing down the hypothesis, primary metric, sample size and decision rule before launch. The cheapest defence available against reinterpreting results after the fact. |
| Predictive Targeting | Personalising on a model score, such as propensity to churn or to buy, rather than on fixed rules. |
| Primary Metric | The single metric the decision hangs on, chosen before launch. More than one primary metric means there is no primary metric. |
| Prior Distribution | The belief about the effect held before seeing the data, expressed as a distribution. Weakly informative priors add stability at little cost; strong priors need justifying, because they can carry a decision on their own. |
| Probabilistic Matching | Inferring that identifiers belong to the same person from behavioural and device signals rather than a shared login. Extends coverage past authenticated users, with an error rate attached. |
| Probability to Beat Control | The posterior probability that a variant’s true rate exceeds the control’s. Intuitive and often misread as a promise of magnitude: 97% probability of winning says nothing about whether the win is worth having. |
| Propensity Model | A model scoring how likely a user is to take an action, used to target offers or to suppress unnecessary discounts. Needs retraining and a holdout, because acting on the score changes the behaviour it was trained on. |
| Proxy Metric | A fast-moving stand-in for a slow outcome that actually matters. Trustworthy only while the relationship to the real outcome holds, and sustained optimisation pressure is exactly what breaks that relationship. |
Q
| Term | Definition |
| QA | See Quality Assurance. |
| Qualitative Research | Non-numeric investigation such as user interviews, session replay and usability testing. It tells you why users behave as they do, which experiments cannot. |
| Quality Assurance | Checking that every variant renders and tracks correctly across the browsers, devices and states present in the audience before traffic starts. A variant broken on one browser looks identical to a losing variant. |
| Quantitative Research | Numeric measurement of behaviour, including analytics and experimentation. It tells you what is happening and by how much, without the reason. |
| Quasi-Experiment | A study with a comparison group but no randomisation, such as difference-in-differences across a staged rollout. The fallback when randomisation is impossible, resting on assumptions that randomisation would have made unnecessary. |
R
| Term | Definition |
| ROI | See Return on Investment. |
| RPV | See Revenue Per Visitor. |
| Ramp-Up | Increasing the share of traffic exposed to a variant in stages. Limits the damage a bad release can do, and the pre-ramp and post-ramp populations are not comparable, so analyse within a consistent allocation window. |
| Randomisation | Assigning units to arms by chance, which balances every confounder in expectation, the unknown ones included. Everything else in the method is bookkeeping around this single act. |
| Recommendation Engine | A system that selects which items to show a user, driven by collaborative filtering, content features, or a learned ranking model. Evaluate it live against a holdout, since offline ranking metrics regularly disagree with online outcomes. |
| Regression to the Mean | The tendency for extreme measurements to be followed by less extreme ones. It is why a segment chosen because it was performing unusually badly appears to improve after any intervention at all, including none. |
| Relative Difference | See Lift. |
| Replication | Running a test again to confirm its result. Rare in practice, and the fastest way to discover how many past wins were noise. |
| Representative Sample | A sample whose composition matches the population it stands for. Online tests get this for free while running, and lose it whenever the test period itself is unusual. |
| Retention Rate | The share of users still active after a defined period. The metric tied most closely to long-run value, and too slow for most tests to move detectably. |
| Return on Investment | Net gain divided by cost. For a testing programme, count the losses prevented as well as the wins shipped, since blocked bad releases are usually most of the value. |
| Revenue Per Visitor | Total revenue divided by visitors, the revenue counterpart to conversion rate. High variance, because it multiplies a rare event by a skewed amount, so it needs far more traffic than conversion rate does. |
| Rollout | Releasing the winning variant to all traffic. Keep measuring afterwards, because the effect at 100% frequently differs from the effect at 50%. |
| Rules-Based Personalisation | Personalisation driven by hand-written if-then rules over segments. Transparent and easy to audit, and it stops scaling as soon as the rules begin to conflict with each other. |
S
| Term | Definition |
| SRM | See Sample Ratio Mismatch. |
| Sample | The set of units actually observed, standing in for the population. Every number in a test report is a property of the sample being used to guess at the population. |
| Sample Ratio Mismatch | A statistically significant deviation between the observed split of traffic and the intended one, for example 49.2% against 50%. It invalidates the test outright rather than merely weakening it, because whatever misallocated users also plausibly biased the outcome. Check it before reading any result. |
| Sample Size | The number of units in the test. Fixed in advance for a classical test, and the single lever that determines what size of effect can be resolved. |
| Sampling Distribution | The distribution of an estimate across hypothetical repeated samples. All frequentist inference is a statement about it, which is why normality of the estimate matters and normality of the raw data does not. |
| Seasonality | Systematic variation by hour, day, week or season. Handled by running whole weeks and avoiding unusual periods, since a test that starts on a Friday and ends on a Tuesday has weighted weekends unequally between its early and late data. |
| Segment | A defined subset of users, such as mobile visitors from paid search. Useful for understanding, dangerous for deciding, because segment-level results are noisier and more numerous than anyone accounts for. |
| Segmentation | Dividing the audience into groups expected to behave differently, the foundation of rules-based personalisation and of segment analysis in tests. |
| Sequential Testing | Any method designed for repeated analysis of accumulating data with error control maintained, through alpha-spending or always-valid inference. It legitimises the monitoring that everyone does anyway. |
| Server-Side Testing | Variants rendered by the server or application before the response reaches the browser. No flicker and no added page weight, at the cost of engineering involvement for every test. |
| Session | A bounded sequence of a user’s activity, usually cut by a period of inactivity. A convenient unit and a poor randomisation unit, since the same person can land in different arms across sessions. |
| Session Replay | Recorded playback of individual user sessions. Excellent for spotting broken interfaces and generating hypotheses, and not evidence of anything at scale. |
| Significance Level | See Alpha-Spending for how it is budgeted, and Nominal Significance Level for the planned value. It is the false positive rate the test is willing to accept. |
| Simpson’s Paradox | When a result reverses direction after the data is split, or after splits are pooled. It appears in tests when the mix of segments differs between arms, which is one more reason sample ratio mismatch is fatal. |
| Single Customer View | One unified, deduplicated record per customer across all systems. The prerequisite for coherent cross-channel personalisation and the thing most organisations do not actually have. |
| Split Test | See A/B Test. |
| Split URL Test | A test that sends traffic to different URLs rather than modifying one page. Suits full redesigns, and it needs care so that the URL difference itself, in bookmarks, analytics and search, does not create the effect. |
| Standard Deviation | The typical distance of observations from their mean, in the metric’s own units. Describes the spread of users, not the precision of the estimate. |
| Standard Error | The standard deviation of the estimate itself, shrinking with the square root of sample size. This is the quantity that sets interval width, so quadrupling traffic halves the uncertainty. |
| Statistical Significance | The finding that observed data would be sufficiently unlikely under the null hypothesis to reject it at the chosen alpha. It speaks to the existence of an effect, never to its size or its business value. |
| Stopping Rule | The pre-declared condition under which a test ends. It must be fixed before launch, because the properties of every statistic in the report depend on it. |
| Stratified Sampling | Splitting the population into strata and sampling within each, ensuring each is properly represented. Related to blocking, and used in analysis to reweight when arms drift apart in composition. |
| Survivorship Bias | Drawing conclusions from only the units that made it to the end, ignoring those that dropped out. It appears whenever exposure is defined at a point that already filters users. |
T
| Term | Definition |
| Targeting | Choosing which users are eligible for a test or a personalised experience. Narrow targeting raises the effect within the audience and cuts the traffic available, so it trades power for relevance. |
| Test Duration | How long a test runs. Driven by required sample size, but with a floor of at least one full business cycle, usually one or two whole weeks, no matter how quickly the sample accumulates. |
| Test Velocity | How many tests a team runs per unit of time. The strongest predictor of programme value, since win rates are low and learning compounds only with volume. |
| Traffic Allocation | The share of eligible traffic entering the test, and how it is split across arms. Equal splits maximise power for a fixed total; uneven splits are a deliberate trade of power for risk containment. |
| Treatment | The changed experience being evaluated, the variant. |
| Trigger Condition | The rule that decides when a user enters the test or sees a personalised experience. Triggering as close as possible to the actual change keeps diluting non-exposed users out of the measurement. |
| Trimmed Mean | A mean computed after discarding a fixed percentage of the extreme values. A pre-declared way to stop a handful of enormous orders dictating the outcome. |
| Twyman’s Law | The observation that any figure that looks interesting or unusual is probably wrong. The correct first response to a surprisingly large win is to check the instrumentation. |
| Two-Sided Test | A test that splits alpha between both directions, detecting harm as well as benefit. The safer default, and the right choice whenever a loss would matter as much as a gain. |
| Type I Error | Rejecting a true null, in other words a false positive. Its rate is alpha, which is the one error rate the analyst sets directly. |
| Type II Error | Failing to reject a false null, in other words a false negative. Its rate is one minus power, and it is controlled indirectly through sample size. |
U
| Term | Definition |
| Unique Visitor | A distinct identifier seen in a period, the usual proxy for a person. It overcounts, since one person on three devices with cleared cookies is several unique visitors. |
| Uplift | See Lift. |
| Uplift Modelling | Modelling the incremental effect of a treatment per user, so targeting goes to those who will respond because of it rather than those who would have converted anyway. The right objective for discounting and retention offers, and much harder to fit than a propensity model. |
| Usability Testing | Watching real users attempt tasks on the interface. Finds problems in a handful of sessions that no amount of traffic will reveal, and it cannot measure how much fixing them is worth. |
| User | The randomisation unit in most online tests, represented by a stable identifier rather than a verified person. |
V
| Term | Definition |
| Validity | Whether the test measures what it claims to. Split into internal validity, meaning the effect is real within the test, and external validity, meaning it generalises beyond it. |
| Vanity Metric | A metric that looks good in a report and drives no decision, such as raw pageviews. Recognisable by the fact that no result would change what anyone does. |
| Variance | The average squared distance from the mean, the raw material of uncertainty. Everything that reduces it, from blocking to CUPED to trimming, buys the same result with less traffic. |
| Variance Reduction | Any technique that lowers the standard error without biasing the estimate, including CUPED, stratification and regression adjustment. The cheapest way to run more tests, since it shortens every test rather than adding traffic. |
| Variant | One of the experiences being compared, usually meaning a treatment rather than the control. |
| Visitor | A counted identifier arriving at the site. See Unique Visitor for the counting caveat. |
W
| Term | Definition |
| Washout Period | A gap between experiments on the same surface, allowing carryover effects to fade before the next test starts. |
| Web Analytics | The measurement of site behaviour through page and event tracking. It describes what happened; only an experiment establishes what caused it. |
| Welch’s t-Test | A t-test that does not assume equal variances across groups. The sensible default for comparing means in online tests, since there is rarely any reason to believe the arms have identical variance. |
| Winner’s Curse | The tendency for the measured effect of a test stopped early, or selected as the best of many, to overstate the true effect. Expect the rollout to underdeliver against the reported lift, particularly after early stopping. |
| Within-Subjects Design | A design in which the same user experiences more than one variant. Powerful in lab settings and normally unworkable online, because users notice the change and the exposures contaminate each other. |
Y
| Term | Definition |
| Year-over-Year | Comparing a period against the same period a year earlier, to control for seasonality. A reporting device, not a causal one, since everything else changed in that year too. |
Z
| Term | Definition |
| Z-Score | The number of standard errors between an observed estimate and the null value. The test statistic behind proportion tests, converted to a p-value through the normal distribution. |
| Z-Test | A test comparing means or proportions using the normal approximation. The standard test for conversion rates at online sample sizes. |
| Zero-Party Data | Data a customer knowingly and deliberately provides, such as stated preferences in a quiz or preference centre. Increasingly the most defensible basis for personalisation as third-party identifiers disappear. |
[…] Glossary: A/B testing and personalisation […]
[…] Glossary: A/B testing and personalisation […]