§3.9.2–3.9.6Multinoulli · Gaussian · Exponential & Laplace · Dirac & Empirical · Mixtures

§3.9.2 The multinoulli distribution

The multinoulli (categorical) distribution multinoulli distribution (Categorical) one discrete variable with k enumerable states, parametrized by p ∈ [0,1]^{k−1}. defined in ch. 3 — open in glossary handles one discrete variable with $k$ finite states, parametrized by $\boldsymbol{p} \in [0,1]^{k-1}$ — the last state’s probability is whatever remains, $1 - \boldsymbol{1}^\top\boldsymbol{p}$ (so we must keep $\boldsymbol{1}^\top\boldsymbol{p} \le 1$). States are usually categories with no numeric value, so expectations and variances of multinoulli variables are rarely needed.

Bernoulli and multinoulli can describe any distribution over their domains — not because they are powerful, but because their domains are simple enough to enumerate. Continuous variables have uncountably many states: any small-parameter description must impose real limits.

§3.9.3 The Gaussian distribution

The most-used distribution over the reals — the normal or Gaussian distribution gaussian distribution N(x; μ, σ²), the bell curve — default for reals via the central limit theorem and max entropy for fixed variance; multivariate form uses covariance Σ or precision matrix. defined in ch. 3 — open in glossary :

Evaluating the PDF squares and inverts $\sigma$ each time; when evaluating repeatedly with different parameters, the precision precision Inverse variance β = 1/σ² (or matrix β = Σ⁻¹): the cheap-to-evaluate parametrization of a Gaussian. defined in ch. 3 — open in glossary $\beta = 1/\sigma^2 \in (0, \infty)$ parametrization $\mathcal{N}(x; \mu, \beta^{-1}) = \sqrt{\beta/2\pi}\, e^{-\frac{1}{2}\beta(x-\mu)^2}$ is cheaper.

In $\mathbb{R}^n$ it becomes the multivariate normal, parametrized by a positive definite symmetric covariance $\boldsymbol{\Sigma}$:

Common simplifications: diagonal covariance, or the isotropic Gaussian (covariance = scalar × identity).

§3.9.4 Exponential and Laplace

Deep learning often wants a sharp point at $x = 0$ — the exponential distribution $p(x; \lambda) = \lambda\,\boldsymbol{1}_{x \ge 0}\,e^{-\lambda x}$ (the indicator function zeroes all negative $x$). To put the sharp peak at an arbitrary $\mu$, use the Laplace distribution laplace distribution Laplace(x; μ, γ): a sharp probability peak at an arbitrary point μ (exponential distribution: sharp point at 0). defined in ch. 3 — open in glossary $\mathrm{Laplace}(x; \mu, \gamma) = \frac{1}{2\gamma} e^{-|x-\mu|/\gamma}$:

§3.9.5 Dirac and empirical distributions

To concentrate all mass at one point, use the Dirac delta function: $p(x) = \delta(x - \mu)$ — zero everywhere except $\mu$, yet integrating to 1. It is not an ordinary function but a generalized function: the limit of functions putting ever less mass everywhere else. Its main job is building the empirical distribution empirical distribution p̂(x) = (1/m) Σ δ(x − x⁽ⁱ⁾): mass 1/m on each training point — what training samples from; maximizes training likelihood. defined in ch. 3 — open in glossary of a dataset:

(For discrete variables no Dirac is needed — the empirical distribution is just a multinoulli with probabilities equal to empirical frequencies.)

§3.9.6 Mixtures of distributions

Richer distributions come from combining simple ones. A mixture distribution mixture distribution P(x) = Σᵢ P(c=i) P(x|c=i): a multinoulli picks which component generates each sample. defined in ch. 3 — open in glossary picks, on each trial, which component generates the sample:

The empirical distribution was already a mixture — one Dirac component per example. The latent variable latent variable A random variable that is never observed directly, like the mixture's component identity c. defined in ch. 3 — open in glossary $c$ relates to $x$ through $P(x, c) = P(x|c)P(c)$; after observing $x$, the updated belief $P(c|x)$ is the posterior posterior probability The updated belief P(c|x) AFTER observing x. defined in ch. 3 — open in glossary (versus the prior prior probability The model's belief P(c) BEFORE observing the data x. defined in ch. 3 — open in glossary $P(c)$) — vocabulary that chapters 5, 13 and 16–20 use constantly.

The star example: the Gaussian mixture model gaussian mixture model Mixture with Gaussian components (means μ⁽ⁱ⁾, covariances Σ⁽ⁱ⁾, priors αᵢ); universal approximator of smooth densities. defined in ch. 3 — open in glossary , components $p(x|c=i) = \mathcal{N}(x; \mu^{(i)}, \Sigma^{(i)})$ — a universal approximator of densities (any smooth density, to any nonzero error, with enough components):

Next: the functions that parametrize these distributions — sigmoid, softplus, Bayes’ rule and information theory.