Calculus of Variations · Continuous Latent Variables · Learned Inference · Wake-Sleep

§19.4.2 Calculus of variations

For discrete latents the variational parameters are numbers; for continuous latents, the optimization is over a function $q$ , and we need calculus of variations — the source of the word “variational”. A functional $J[f]$ maps a whole function to a scalar (like the entropy of a density); its functional derivative generalizes the gradient:

The functional derivative (eq 19.46)

\frac{\delta}{\delta f(\boldsymbol{x})} \int g(f(\boldsymbol{x}), \boldsymbol{x})\, d\boldsymbol{x} = \frac{\partial}{\partial y} g(f(\boldsymbol{x}), \boldsymbol{x})

$\frac{\delta}{\delta f(\boldsymbol{x})}$	the functional derivative w.r.t. the VALUE of f at the point x — think of f as a vector with uncountably many entries indexed by x	per-point
$\frac{\partial}{\partial y} g$	just the ordinary partial derivative of the integrand w.r.t. its function-value slot. Optimize a functional by setting this to 0 at every x	per-point

(The general form, allowing $g$ to depend on $f$ ‘s derivatives too, is the Euler–Lagrange equation ; this book doesn’t need it.) The classic payoff: which density maximizes entropy at a fixed mean and variance? Build the Lagrangian $H[p]$ + constraints (∫p = 1, mean = µ, variance = σ²) and set $\delta/\delta p = 0$ (eqs 19.50–19.52); the algebra derives $p(x) = \mathcal{N}(x; \mu, \sigma^2)$ (eq 19.54) — we never assumed a Gaussian. See it: every curve below has the same variance, but the Gaussian wins on entropy:

Calculus of variations, made visible. A generalized-normal family exp(−|x/α|^β), all rescaled to the SAME variance σ², swept by the shape exponent β. The entropy H[p] peaks exactly at β=2 — the Gaussian. This is why we assume a Gaussian when we know only the variance: it commits to the least additional structure.

β = 1.00σ = 1.00

H[p] = 1.360

Gaussian max = 1.419

Every curve here has the same variance σ², yet their entropies differ — and the entropy H[p] peaks exactly at the Gaussian (β=2). Calculus of variations proves this: solving δ/δp = 0 with a fixed-variance constraint derivesp(x)=N(µ,σ²). It's why we assume a Gaussian when we know only the variance — the least-committal choice.

This is the max-entropy justification for the normal distribution. (One caveat: the method finds stationary functions, so it misses degenerate solutions like Dirac mixtures, which don’t live in function space — those must be found by guessing and verifying.)

§19.4.3 Continuous latent variables

Happily, practitioners rarely solve a variational problem by hand. There is a general mean-field fixed-point equation giving the optimal factor $q(h_i)$ for any model:

The universal mean-field update (eq 19.56)

\tilde{q}(h_i \mid \boldsymbol{v}) = \exp\left(\mathbb{E}_{\boldsymbol{h}_{-i} \sim q(\boldsymbol{h}_{-i}\mid\boldsymbol{v})}\left[\log \tilde{p}(\boldsymbol{v}, \boldsymbol{h})\right]\right)

$\mathbb{E}_{\boldsymbol{h}_{-i} \sim q}$	average the joint log-probability over the OTHER latents, using their current q — hold q(h_j) fixed for j ≠ i	expectation
$\tilde{q}(h_i\mid\boldsymbol{v})$	the unnormalized optimal factor; normalize it to get q(hᵢ\|v). Iterating over i is the fixed-point loop	unnormalized density

Its power is that it hands you not just an iterative update but the functional form of the optimum. For a toy model $p(\boldsymbol{h})=\mathcal{N}(0,I)$ , $p(v\mid\boldsymbol{h})=\mathcal{N}(w^\top\boldsymbol{h}, 1)$ , carrying out the expectation (eqs 19.57–19.67) reveals $\tilde{q}$ has a Gaussian form — derived, not assumed. You can then treat its mean and variance as parameters and optimize them however you like.

§19.4.4 Learning and inference feed back

Approximate inference doesn’t just approximate — it reshapes the model. Since variational learning increases $\mathbb{E}_{\boldsymbol{h}\sim q}[\log p(\boldsymbol{v},\boldsymbol{h})]$ (eq 19.68), it raises $p$ where $q$ is large and lowers it where $q$ is small:

The self-fulfilling prophecy

Train with a unimodal approximate posterior and the model’s TRUE posterior drifts toward being unimodal — the approximation makes its own assumptions come true. This makes the harm of a variational approximation hard to measure: you might find $\mathcal{L} \approx \log p(\boldsymbol{v})$ at the learned $\boldsymbol{\theta}$ and conclude “the approximation is accurate,” yet a better inference algorithm could have reached a $\boldsymbol{\theta}^*$ with much higher likelihood that your simple $q$ can never fit. The only sure way to detect it is to already have a superior learner for comparison.

§19.5 Learned approximate inference

Iterating fixed-point equations for every example is slow. Since inference is just the map $f: \boldsymbol{v} \mapsto q^{*} = \arg\max_q \mathcal{L}(\boldsymbol{v}, q)$ , we can learn it — approximate that whole optimization with a single neural network $\hat{f}(\boldsymbol{v}; \boldsymbol{\theta})$ that outputs the approximate posterior in one forward pass. This is learned approximate inference .

§19.5.1 Wake-sleep

The obstacle: there are no supervised targets — given $\boldsymbol{v}$ we don’t know the right $\boldsymbol{h}$ . The wake-sleep algorithm (Hinton et al. 1995) manufactures them by sampling from the model:

Wake-sleep — the “sleep” phase draws (h, v) pairs from the model by ancestral sampling (h → v), giving a supervised dataset on which the inference network learns the reverse map v → h. Drawback: it only ever sees v’s the model itself generates, so early in training those don’t resemble real data.

A second theory of dreaming

Chapter 18 floated dreams as negative-phase samples for the partition function; wake-sleep offers another reading — dreams could be model samples for training an inference network. It fits the schedule better: animals are awake (improving $q$ ) for hours, then asleep (the generative model held fixed) for hours, which is awkward for Monte Carlo partition-function training but natural for maximizing $\mathcal{L}$ . Purely speculative, of course.

Next: other forms of learned inference — from DBM inference nets to the variational autoencoder.

Check yourself — calculus of variations & learned inference

1.What does calculus of variations let us do that ordinary calculus does not, and why is it needed here?

2.The calculus-of-variations entropy example derives which density, and what does that justify?

3.What does the universal mean-field fixed-point equation q̃(hᵢ|v) = exp(E_{h₋ᵢ∼q}[log p̃(v,h)]) provide beyond an iterative update?

4.How does approximate inference create a 'self-fulfilling prophecy' during learning?

5.How does the wake-sleep algorithm obtain training targets for its inference network, and what is the drawback?

5 questions