Why Probability? · Random Variables · Probability Distributions

Probability theory is the mathematical framework for representing and reasoning about uncertain statements. AI uses it two ways: the laws of probability tell us how systems should reason (so we design algorithms to compute or approximate the right expressions), and probability lets us analyze proposed systems theoretically. Information theory — the second half of this chapter — then quantifies how much uncertainty a distribution contains.

Aside — how to read this chapter

Comfortable with both subjects? Skip everything except §3.14, the graph language the rest of the book uses for structured probabilistic models. Brand new? This chapter suffices for deep learning research, but a dedicated text (Jaynes, 2003) is recommended too.

§3.1 Why probability?

Most of computer science lives in a deterministic world — a CPU executes every instruction flawlessly, and software rarely plans for hardware error. Machine learning is the opposite: it always deals with uncertain, and sometimes genuinely stochastic, quantities. Beyond mathematical tautologies, it is hard to name any proposition that is absolutely true or event absolutely guaranteed. Uncertainty enters through exactly three doors:

Even when the true rule is deterministic and representable, a simple uncertain rule often beats a complex certain one: “most birds fly” is cheap, broadly useful and robust; “birds fly, except very young birds, sick or injured birds, flightless species including the cassowary, ostrich and kiwi…” is expensive to build, maintain and communicate — and still brittle.

Two meanings of “probability,” one set of rules

Frequentist probability : the rate at which a repeatable event occurs — probability $p$ of a poker hand means proportion $p$ of infinitely many draws.
Bayesian probability : a degree of belief about a non-repeatable proposition — “this patient has a 40 % chance of flu” cannot mean replicas of the patient; 1 is absolute certainty of flu, 0 absolute certainty of none.

Remarkably, listing the common-sense properties we expect of uncertain reasoning forces both kinds to obey exactly the same axioms (Ramsey, 1926): the poker formulas and the diagnosis formulas are one and the same.

What this really says

Probability is logic extended to uncertainty. Logic derives which propositions must be true given that others are; probability derives how likely a proposition is given the likelihoods of others. That is why one calculus serves frequencies and beliefs alike.

§3.2 Random variables

A random variable is a variable that takes values randomly — written in plain typeface ( $\mathrm{x}$ ), with its possible values in italic ( $x_1, x_2$ ); vector-valued random variables are upright bold ( $\mathbf{x}$ ) with values $\boldsymbol{x}$ (the typography is on the notation page). On its own a random variable just lists possible states; it must be coupled with a probability distribution saying how likely each state is. Variables are discrete (finitely or countably many states — not necessarily numbers; named categories count) or continuous (real-valued).

§3.3 Probability distributions

§3.3.1 Discrete variables: probability mass functions

A probability mass function (PMF) $P$ maps each state to its probability; $P(x)$ means $P(\mathrm{x} = x)$ , and $\mathrm{x} \sim P(\mathrm{x})$ declares the distribution. PMFs over several variables at once are joint distributions: $P(\mathrm{x} = x, \mathrm{y} = y)$ . Three properties make a function a PMF:

The PMF contract

\text{domain}(P) = \text{states}(\mathrm{x}); \quad \forall x,\; 0 \le P(x) \le 1; \quad \sum_{x \in \mathrm{x}} P(x) = 1

$\text{domain}$	P must cover every possible state of x	completeness
$0 \le P \le 1$	nothing is less likely than impossible or more likely than guaranteed	bounds
$\sum P = 1$	"normalized" — without this, the probability of one-of-many events could exceed 1	normalization

Worked example — the uniform die

A discrete $\mathrm{x}$ with $k$ states is uniform when $P(\mathrm{x} = x_i) = 1/k$ (eq 3.1). For a fair die, $k = 6$ : each face has $P = 1/6 \approx 0.167 > 0$ , and $\sum_i 1/k = k/k = 1$ (eq 3.2) — both requirements check out.

Discrete uniform PMF — slide k and watch every bar shrink so the total stays exactly 1

k6.00

every state gets P = 1/k = 0.167; the 6 bars sum to exactly 1

§3.3.2 Continuous variables: probability density functions

Continuous variables have uncountably many states, so no state can carry finite mass. A probability density function (PDF) $p$ instead satisfies:

The PDF contract — note what is MISSING

\text{domain}(p) = \text{states}(\mathrm{x}); \quad \forall x,\; p(x) \ge 0; \quad \int p(x)\,dx = 1

$p(x) \ge 0$	densities are non-negative — but there is NO requirement p(x) ≤ 1	lower bound only
$p(x)\,\delta x$	the actual probability of landing in an infinitesimal region of volume δx	mass = density × volume
$\int_S p(x)\,dx$	the probability that x lies in a set S — integrate the density	probability of regions

What this really says

A density is probability per unit length, not probability. Any single point has probability zero; only regions accumulate mass. And just as a physical object can have density above 1 g/cm³, a PDF can rise above 1 — squeeze the uniform below and watch it happen.

The continuous uniform $u(x; a, b)$ puts $1/(b-a)$ on $[a, b]$ and 0 outside (written $\mathrm{x} \sim U(a, b)$ ). The " $;$ " reads “parametrized by”: $x$ is the argument; $a$ and $b$ define the particular function.

Continuous uniform PDF — drag b toward a: the density exceeds 1, the area never does

a0.00b1.00

height 1/(b−a) = 1.00; shaded area = 1

Next: combining, splitting and conditioning distributions — marginals, conditionals and the chain rule.

Check yourself — why probability, PMFs and PDFs

1.The Monty Hall problem illustrates which source of uncertainty?

2.Why does the book prefer 'most birds fly' over the exhaustive exception-listing rule?

3.A doctor says a patient has a 40% chance of flu. What kind of probability is this, and why do the usual formulas still apply?

4.In the continuous-uniform widget you drag b until b − a = 0.5. The density reads 2.0. Is this a bug?

5.What is a random variable, before any distribution is attached?

5 questions