§3.10–3.13Useful Functions · Bayes' Rule · Continuous Details · Information Theory

§3.10 Useful properties of common functions

Two functions recur constantly when building probability distributions. The logistic sigmoid logistic sigmoid σ(x) = 1/(1+e⁻ˣ) ∈ (0,1): produces Bernoulli φ; saturates at both extremes; σ' = σ(1−σ). defined in ch. 3 — open in glossary $\sigma(x) = 1/(1 + e^{-x})$ has range $(0, 1)$ — exactly a Bernoulli $\phi$ — but saturates at both extremes (flat, insensitive to small input changes). The softplus softplus ζ(x) = log(1+eˣ) ∈ (0,∞): softened max(0,x); produces σ/β params; ζ' = σ. defined in ch. 3 — open in glossary $\zeta(x) = \log(1 + e^x)$ has range $(0, \infty)$ — a valid $\sigma$ or $\beta$ for a Gaussian — and is a smoothed (“softened”) version of the positive part $x^+ = \max(0, x)$:

The identities the book suggests memorizing (eqs 3.33–3.41) — click a row for why it matters:

§3.11 Bayes’ rule

Knowing $P(\mathrm{y}|\mathrm{x})$ but needing $P(\mathrm{x}|\mathrm{y})$ happens constantly; with the prior $P(\mathrm{x})$, Bayes’ rule bayes rule P(x|y) = P(x)P(y|x)/P(y), with P(y) = Σ_x P(y|x)P(x) — invert a conditional using the prior. defined in ch. 3 — open in glossary inverts the conditional:

(It follows directly from the conditional definition; Bayes found a special case, Laplace the general form.)

§3.12 Technical details of continuous variables

Rigorous continuous probability needs measure theory; the book keeps just three take-aways. (1) Pathological sets (fractal-like, built from the infinite precision of the reals) can break naive integration — measure theory characterizes the safe sets, and everything integrated in this book is safe. (2) A set of measure zero measure zero A set occupying no volume (points, lines in ℝ²); 'almost everywhere' = except on such a set. defined in ch. 3 — open in glossary occupies no volume (points, lines in $\mathbb{R}^2$, countable unions of such); a property holding almost everywhere holds except on such a set — good enough for most purposes. (3) The change-of-variables trap: for $y = g(x)$ with invertible smooth $g$, it is tempting but wrong to write $p_y(y) = p_x(g^{-1}(y))$.

Preserving $|p_y(g(x))\,dy| = |p_x(x)\,dx|$ gives the correct rule — in higher dimensions the stretch factor is §2.11’s determinant, applied to the Jacobian matrix $J_{i,j} = \partial x_i / \partial y_j$:

§3.13 Information theory

Born for optimal codes over noisy channels, information theory gives ML its tools for characterizing distributions and comparing them. The seed intuition: learning that an unlikely event occurred is more informative (“solar eclipse this morning” ≫ “the sun rose”). Three requirements — certain events carry zero information, less likely means more information, independent events add — force one definition:

Self-information self-information I(x) = −log P(x): rare events carry more information; units nats (base e) or bits (base 2). defined in ch. 3 — open in glossary covers one outcome; the Shannon entropy shannon entropy H(x) = E[I(x)] = −E[log P(x)]: expected information — the uncertainty of a whole distribution; max for uniform. defined in ch. 3 — open in glossary is its expectation — the uncertainty of the whole distribution, and the lower bound on average code length for symbols drawn from it: $H(\mathrm{x}) = \mathbb{E}_{\mathrm{x} \sim P}[I(x)] = -\mathbb{E}[\log P(x)]$ (continuous case: differential entropy; convention $0 \log 0 = 0$).

To compare two distributions over the same variable, the Kullback-Leibler divergence kullback-leibler divergence D_KL(P‖Q) = E_{x∼P}[log P − log Q] ≥ 0, zero iff P = Q; asymmetric — NOT a distance. defined in ch. 3 — open in glossary :

The asymmetry has real consequences (fig 3.6): approximating a bimodal $p$ with a single Gaussian $q$,

Finally, the cross-entropy cross-entropy H(P,Q) = H(P) + D_KL(P‖Q) = −E_{x∼P} log Q(x); minimizing over Q ≡ minimizing the KL. defined in ch. 3 — open in glossary $H(P, Q) = H(P) + D_{\mathrm{KL}}(P \| Q) = -\mathbb{E}_{\mathrm{x} \sim P} \log Q(x)$: minimizing it over $Q$ is the same as minimizing the KL (the omitted $H(P)$ doesn’t involve $Q$) — which is why “cross-entropy loss” and “KL minimization” name one procedure in ch. 5–6.