§3.4–3.9.1Marginals · Conditionals · Chain Rule · Independence · Expectation, Variance, Covariance · Bernoulli

§3.4 Marginal probability

Given a joint distribution over several variables, the distribution of just a subset is the marginal probability marginal probability The distribution of a subset of variables: P(x) = Σ_y P(x,y) (sum rule) or the integral analogue. defined in ch. 3 — open in glossary distribution, found with the sum rule:

(Continuous variables integrate instead: $p(x) = \int p(x, y)\,dy$.) The etymology is literal — try it:

§3.5 Conditional probability

The probability of an event given that another has happened is a conditional probability conditional probability P(y|x) = P(y,x)/P(x), defined when P(x) > 0; NOT the same as computing an intervention's consequences. defined in ch. 3 — open in glossary :

In the widget above, clicking a row performs exactly this renormalization.

§3.6 The chain rule of conditional probabilities

Any joint distribution factors into one-variable conditionals — the chain rule chain rule of probability Any joint factors into one-variable conditionals: P(x⁽¹⁾…x⁽ⁿ⁾) = P(x⁽¹⁾) Π P(x⁽ⁱ⁾|x⁽¹⁾…x⁽ⁱ⁻¹⁾). defined in ch. 3 — open in glossary (or product rule): $P(\mathrm{x}^{(1)}, \dots, \mathrm{x}^{(n)}) = P(\mathrm{x}^{(1)}) \prod_{i=2}^n P(\mathrm{x}^{(i)} \mid \mathrm{x}^{(1)}, \dots, \mathrm{x}^{(i-1)})$. It falls straight out of the definition, applied repeatedly:

§3.7 Independence and conditional independence

Two variables are independent independence x⊥y: the joint factorizes p(x,y) = p(x)p(y); stronger than zero covariance (excludes nonlinear relations too). defined in ch. 3 — open in glossary ($\mathrm{x} \bot \mathrm{y}$) when their joint factorizes: $p(x, y) = p(x)\,p(y)$ for every pair of values — the grid becomes an outer product of its margins (flip the widget above to its independent version: same marginals, different joint!). They are conditionally independent conditional independence x⊥y|z: the joint over x,y factorizes given every value of z. defined in ch. 3 — open in glossary given $\mathrm{z}$ ($\mathrm{x} \bot \mathrm{y} \mid \mathrm{z}$) when the factorization holds within every slice $z$.

§3.8 Expectation, variance and covariance

The expectation expectation E_{x∼P}[f(x)] = Σ P(x)f(x) (or integral): the average of f under P; linear. defined in ch. 3 — open in glossary (expected value) of $f(x)$ under $P$ is the average value $f$ takes when $x$ is drawn from $P$:

The variance variance E[(f(x) − E[f(x)])²]: spread around the mean; its square root is the standard deviation. defined in ch. 3 — open in glossary $\mathrm{Var}(f(x)) = \mathbb{E}\big[(f(x) - \mathbb{E}[f(x)])^2\big]$ measures spread around the mean (low variance = values cluster); its square root is the standard deviation. The covariance covariance E[(f(x)−E f)(g(y)−E g)]: linear co-variation and scale; correlation normalizes the scale away. defined in ch. 3 — open in glossary $\mathrm{Cov}(f(x), g(y)) = \mathbb{E}\big[(f(x) - \mathbb{E}[f(x)])(g(y) - \mathbb{E}[g(y)])\big]$ measures linear co-variation (and carries the variables’ scales; correlation normalizes the scales away).

Covariance is NOT dependence. Independence ⟹ zero covariance, and nonzero covariance ⟹ dependence — but zero covariance does not imply independence, because covariance only sees linear relationships:

For a random vector $\mathbf{x} \in \mathbb{R}^n$, the covariance matrix covariance matrix For random vector x ∈ ℝⁿ: Cov(x)_{i,j} = Cov(xᵢ, xⱼ); variances on the diagonal. defined in ch. 3 — open in glossary collects everything: $\mathrm{Cov}(\mathbf{x})_{i,j} = \mathrm{Cov}(x_i, x_j)$, with the variances $\mathrm{Var}(x_i)$ on the diagonal — the object the multivariate Gaussian is built around.

§3.9.1 The Bernoulli distribution

The Bernoulli distribution bernoulli distribution Single binary variable with P(x=1) = φ; mean φ, variance φ(1−φ). defined in ch. 3 — open in glossary covers a single binary variable via one parameter $\phi \in [0, 1]$: $P(\mathrm{x} = 1) = \phi$, $P(\mathrm{x} = 0) = 1 - \phi$ — compactly $P(\mathrm{x} = x) = \phi^x (1-\phi)^{1-x}$ — with $\mathbb{E}[\mathrm{x}] = \phi$ and $\mathrm{Var}(\mathrm{x}) = \phi(1 - \phi)$: