§18.0–18.2The Log-Likelihood Gradient · Contrastive Divergence · Stochastic Maximum Likelihood

§18.0 The problem

An undirected model gives an unnormalized $\tilde{p}(\boldsymbol{x}; \boldsymbol{\theta})$; the real distribution needs the partition function partition function Z = Σ/∫ p̃(x): the normalizer of an undirected model (term borrowed from statistical physics); usually intractable in deep learning (ch18) and can fail to exist if the integral diverges. defined in ch. 16 — open in glossary $Z(\boldsymbol{\theta}) = \sum_{\boldsymbol{x}} \tilde{p}$ (or $\int \tilde{p}\,d\boldsymbol{x}$), which is intractable for most interesting models (§16.2.3). Some deep models are designed to have a tractable $Z$, or to be used without ever computing $p(\boldsymbol{x})$ (chapter 20); this chapter is about the models that confront the intractable $Z$ head-on — how to train and evaluate them anyway.

§18.1 The log-likelihood gradient splits in two

Maximum-likelihood learning is hard here because $Z$ depends on the parameters. The gradient splits into two named pieces:

The negative phase looks hopeless — $Z$ is an intractable sum — but it has a beautiful form. Differentiating $\log Z$ (eqs 18.5–18.13, using $\tilde{p} = \exp(\log\tilde{p})$) collapses to an expectation over the model’s own samples:

In energy terms ($\log\tilde{p} = -E$), the two phases are a tug of war: the positive phase positive phase The ∇θ log p̃(x) term of the undirected log-likelihood gradient: raise the unnormalized probability (lower the energy) on training examples. defined in ch. 18 — open in glossary pushes the energy down on training data; the negative phase negative phase The −∇θ log Z = −E_{x∼p}[∇θ log p̃(x)] term: lower the unnormalized probability (raise the energy) on samples drawn from the model; the difficult term. defined in ch. 18 — open in glossary pushes the energy up on samples drawn from the model. When $p_\text{model} = p_\text{data}$, every up-push is matched by a down-push and the gradient vanishes — training is done. Drive it yourself:

The model samples of the negative phase are the points the model believes in most strongly — its wrong beliefs, often called “hallucinations” or fantasy particles fantasy particles The model's own samples used in the negative phase ('hallucinations'): points the model believes in strongly, whose probability the negative phase pushes down. defined in ch. 18 — open in glossary . (The positive/negative split has even been floated as an account of dreaming: follow $\nabla\log\tilde{p}$ awake, its negative asleep.)

§18.2 Making the negative phase affordable

The naive recipe (Algorithm 18.1) estimates eq 18.15 by burning in a fresh set of Markov chains from random initialization at every gradient step — far too expensive. Every practical method is a cheaper way to initialize those negative chains so they need less burn-in:

A crucial bonus: because these methods yield an estimate of $\nabla_\theta \log Z$ separately, you can bolt the negative-phase gradient onto any positive- phase method — even one that only lower-bounds $\tilde{p}$. Most of the other techniques in this chapter (next) cannot compose with a lower-bounded positive phase. Next: methods that sidestep Z entirely — pseudolikelihood, score matching, and noise-contrastive estimation.