The Language of Uncertainty

Probability is the branch of mathematics concerned with measuring how likely events are to occur. It underpins statistical inference, decision-making under uncertainty, and a wide range of applied disciplines from engineering reliability to financial modelling.

An experiment is any process whose outcome cannot be predicted with certainty. The sample space, written S, is the complete set of all possible outcomes. An event is any subset of the sample space, a collection of outcomes whose occurrence we wish to measure.

Rolling a fair six-sided die illustrates these concepts directly. The sample space is S = {1, 2, 3, 4, 5, 6}, giving six outcomes. The event “score greater than 4” is the subset {5, 6}, and the event “even score” is {2, 4, 6}.

Probability When Outcomes Are Equally Likely

When all N outcomes in a sample space are equally likely, the probability of event A containing n favourable outcomes is:

Probability always satisfies 0 ≤ P(A) ≤ 1. An impossible event has probability 0; a certain event has probability 1.

For the die, P(“even score”) = 3/6 = 0.5 and P(“score greater than 4”) = 2/6 ≈ 0.333. An even score is therefore more likely than a score above 4.

The Relative Frequency Interpretation

Many real-world experiments do not produce equally likely outcomes. A biased coin, a manufacturing process, or a medical test each assigns different probabilities to different outcomes. In such settings, probability is understood as the limiting relative frequency. If an experiment is repeated F times independently and event A occurs f times, then:

This means that with enough repetitions, the observed proportion of occurrences converges to the true probability. A biased coin tossed 1,000 times that lands heads 620 times yields an estimated probability of 0.62. The estimate improves as F increases and will not equal 0.5 because the coin is not fair.

The Three Axioms of Probability

The entire structure of probability theory rests on three foundational rules. The first is that probabilities are non-negative: for any event A, P(A) ≥ 0. The second is that the probability of the entire sample space equals 1: P(S) = 1, meaning some outcome must always occur.

The third axiom concerns mutually exclusive events, meaning events that cannot both occur simultaneously. If A₁, A₂, …, Aₙ are mutually exclusive, then:

A collection of events is collectively exhaustive if at least one of them must occur. For a die roll, A = {2, 4, 6} and B = {1, 3, 5} are both mutually exclusive and collectively exhaustive. By the third axiom, P(A ∪ B) = P(A) + P(B) = 0.5 + 0.5 = 1, consistent with the second axiom.

Combining Events with Set Notation

Events can be combined in several ways. The union A ∪ B is the event that at least one of A or B occurs. The intersection A ∩ B is the event that both A and B occur. The complement A^c is the event that A does not occur. Conditional notation A | B, covered later, describes the probability of A given that B is known to have occurred.

These operations become concrete with a worked example. Consider the sample space S = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, representing the possible totals when two dice are rolled. Let A = even totals, B = totals below 8, and C = totals from 5 to 9. Then A = {2, 4, 6, 8, 10, 12}, B = {2, 3, 4, 5, 6, 7}, and C = {5, 6, 7, 8, 9}.

The intersection A ∩ B = {2, 4, 6} captures even totals below 8. The intersection A ∩ C = {6, 8} captures even totals in C. The set A^c ∩ B = {3, 5, 7} captures odd totals below 8. The complement (A ∪ B ∪ C)^c = {11}, the only total belonging to none of the three events.

The Additive Law

The probability that at least one of two events occurs is given by the additive law:

The term P(A ∩ B) is subtracted because any outcome in the intersection would otherwise be counted twice. For mutually exclusive events where A ∩ B is empty, the law simplifies to P(A ∪ B) = P(A) + P(B). The complement rule follows directly:

Suppose a machine component can fail in two distinct ways, with P(D₁) = 0.01, P(D₂) = 0.05, and P(D₁ ∩ D₂) = 0.001. The probability the component is defective in at least one way is:

The probability the component is fault-free is therefore 1 − 0.059 = 0.941.

Independence and the Multiplicative Law

Two events are independent if the occurrence of one provides no information about the probability of the other. The formal condition is:

Independence is frequently confused with mutual exclusivity, but the two are fundamentally different. Mutually exclusive events cannot both occur, so knowing one happened makes the other impossible. Unless at least one event has probability zero, mutually exclusive events are not independent.

Rolling two fair dice produces independent events. The probability of a six on each die is 1/6, and because the dice do not influence each other:

Conditional Probability

The conditional probability of A given that B has occurred is defined as:

Symmetrically:

Rearranging either expression gives P(A ∩ B) = P(A | B) · P(B) = P(B | A) · P(A). The independence condition from the previous section can be stated equivalently as P(A | B) = P(A): knowing B occurred does not change the probability of A.

Consider a cohort of 144 students in which 18 study Mathematics, 53 are female, and 4 are both female and Mathematics students. To check independence, compare P(A) · P(F) with P(A ∩ F). The product P(A) · P(F) = (18/144)(53/144) ≈ 0.046 does not equal P(A ∩ F) = 4/144 ≈ 0.028, so gender and subject choice are not independent. The conditional probabilities are:

Approximately 7.5% of female students study Mathematics, while 22.2% of Mathematics students are female. The asymmetry between these two values reinforces why the direction of conditioning matters.

Bayes’ Formula

Bayes’ formula reverses the direction of conditioning. When P(B | A), P(A), and P(B) are all known, it provides P(A | B):

This result follows directly from the definition of conditional probability applied twice. Suppose P(A) = 0.3, P(B | A) = 0.6, and P(B) = 0.4. Applying Bayes’ formula:

Observing B raises the probability of A from 0.30 to 0.45, reflecting that B is more likely when A is true than in general.

The Total Probability Formula

When the sample space can be divided into complementary events B and B^c — mutually exclusive and collectively exhaustive — the probability of any event A can be decomposed as:

More generally, if B₁, B₂, …, Bₙ partition the sample space into n mutually exclusive and collectively exhaustive events:

This formula is useful whenever A can arise through several distinct routes and the probability of each route is known. Suppose a factory’s output comes 60% from Machine 1 and 40% from Machine 2, with Machine 1 producing a 2% defect rate and Machine 2 a 5% defect rate. The overall probability that a randomly selected item is defective is:

The General Bayes’ Formula

Combining the total probability formula with the definition of conditional probability yields the general form of Bayes’ formula. For a partition B₁, …, Bₙ and an observed event A:

The denominator is simply P(A) as computed by the total probability formula. General Bayes’ formula is particularly valuable when an observation has been made and the goal is to identify which of several possible causes is most likely.

Returning to the factory example: given that an item is defective, what is the probability it came from Machine 1?

Despite Machine 1 supplying 60% of total output, only 37.5% of defective items originate there. Machine 1’s lower defect rate means it contributes fewer defects per unit of output than Machine 2.

Probability Trees

A probability tree provides a structured way to organise multi-stage probability problems. Each branch represents one outcome of one stage, and branch probabilities are conditional on having reached that node. The probability of any path from root to leaf equals the product of all branch probabilities along that path.

Four rules govern probability trees. Probabilities on all branches leaving any single node must sum to 1. The probability of reaching any leaf equals the product of branch probabilities along its path. The probability of any event is the sum of leaf probabilities for all paths where that event occurs. Conditional probabilities are computed by dividing the relevant leaf probability by the sum of all leaf probabilities where the conditioning event holds.

Consider a fault-detection test applied to a system that can fail at one of three sites: A with probability 0.5, B with probability 0.2, or C with probability 0.3. The test correctly identifies the fault location with probability 0.7 for site A, 0.6 for site B, and 0.8 for site C. When it gives an incorrect result, errors are split equally between the remaining two sites. This gives the following conditional probabilities: for site A, P(says A | A) = 0.70 and P(says B | A) = P(says C | A) = 0.15; for site B, P(says B | B) = 0.60 and each error probability is 0.20; for site C, P(says C | C) = 0.80 and each error probability is 0.10.

Multiplying each pair of branch probabilities gives nine joint probabilities:

True site	Test diagnoses	Joint probability
A (0.5)	A	0.5 × 0.70 = 0.350
A (0.5)	B	0.5 × 0.15 = 0.075
A (0.5)	C	0.5 × 0.15 = 0.075
B (0.2)	A	0.2 × 0.20 = 0.040
B (0.2)	B	0.2 × 0.60 = 0.120
B (0.2)	C	0.2 × 0.20 = 0.040
C (0.3)	A	0.3 × 0.10 = 0.030
C (0.3)	B	0.3 × 0.10 = 0.030
C (0.3)	C	0.3 × 0.80 = 0.240

The nine values sum to 1.0, confirming the tree is correctly specified. The probability that the test diagnoses site A — regardless of where the fault actually is — is the sum of all leaves where the test says A: 0.350 + 0.040 + 0.030 = 0.420.

Given that the test has diagnosed site A, the probability that the fault is actually at C is:

The probability that the fault is not at B, given the same diagnosis, is:

The tree makes visible what Bayes’ formula computes algebraically. Each leaf represents a hypothesis-observation pair, and conditioning simply restricts attention to the subset of leaves where the observation holds.

A workbook of 10 exercises for you to practice: https://datalad.co.uk/probability-theory-workbook-10-exercises-with-full-solutions/

See you soon.