§16.5–16.7.1Learning Dependencies · Inference · The Deep Learning Approach · the RBM

§16.5 Learning about dependencies

A good generative model must capture the distribution over the visible variables $\boldsymbol{v}$, whose elements are usually highly dependent on one another. There are two ways to model those dependencies. The traditional way is structure learning structure learning Searching (usually greedily) for a good graph structure by proposing edge add/removes and scoring accuracy vs complexity; deep learning avoids it by using latent variables instead. defined in ch. 16 — open in glossary : search — usually greedily — over graph structures, training and scoring each (accuracy minus a complexity penalty), adding and removing edges. The deep-learning way is to introduce latent latent variable A random variable that is never observed directly, like the mixture's component identity c. defined in ch. 3 — open in glossary (“hidden”) variables $\boldsymbol{h}$ and connect every visible to them, so that any two visibles interact indirectly through $\boldsymbol{h}$ — no direct visible–visible edges, no discrete structure search:

Latent variables earn their keep twice over. Beyond capturing $p(\boldsymbol{v})$ cheaply, $\boldsymbol{h}$ is itself an alternative representation of $\boldsymbol{v}$: as with the mixture of Gaussians (whose latent component index classifies the input) or sparse coding, the learned $\mathbb{E}[\boldsymbol{h} \mid \boldsymbol{v}]$ or $\arg\max_{\boldsymbol{h}} p(\boldsymbol{h}, \boldsymbol{v})$ makes an excellent feature vector — feature learning is latent- variable learning.

§16.6 Inference and approximate inference

We constantly need to answer questions like “given these visibles, what are the latents?” — to extract features $\mathbb{E}[\boldsymbol{h}\mid\boldsymbol{v}]$, or to train by maximum likelihood, which needs $p(\boldsymbol{h}\mid\boldsymbol{v})$:

For most interesting deep models this inference is intractable, even with a graph: the graphs used in deep learning are expressive but not restrictive enough to permit efficient exact inference.

In deep learning that approximation is almost always variational inference: replace the true posterior $p(\boldsymbol{h}\mid\boldsymbol{v})$ with a tractable $q(\boldsymbol{h}\mid\boldsymbol{v})$ chosen as close to it as possible — the whole subject of chapter 19.

§16.7 The deep-learning approach to graphical models

Deep learning uses the same graphical-model tools as everyone else, but makes systematically different design choices — so different that the resulting models have their own flavor:

§16.7.1 Example: the restricted Boltzmann machine

The restricted Boltzmann machine restricted boltzmann machine (rbm) An energy-based model with binary v, h and energy −bᵀv − cᵀh − vᵀWh, with NO intra-layer edges → factorial p(h|v) & p(v|h) with sigmoid conditionals → efficient block Gibbs sampling; the canonical deep-learning graphical model. defined in ch. 16 — open in glossary (RBM), or harmonium (Smolensky, 1986), is the quintessential deep-learning graphical model. It is not itself deep — it has a single latent layer — but it is the building block of many deeper models (chapter 20), and it exemplifies every practice above: units in layers, connectivity by a matrix, relatively dense, designed for efficient Gibbs sampling, latent semantics left to the training algorithm. It is an energy-based model energy-based model An undirected model written p̃(x)=exp(−E(x)), guaranteeing p̃>0 and allowing unconstrained learning of the energy function E. defined in ch. 16 — open in glossary with binary visible and hidden units:

The energy has no visible–visible or hidden–hidden term — that is the “restricted” (a general Boltzmann machine allows arbitrary connections). Drawn as a Markov network it is bipartite:

Because no unit shares a layer with a unit it depends on, the layer-wise conditionals factorize — $p(\boldsymbol{h}\mid\boldsymbol{v}) = \prod_i p(\mathrm{h}_i\mid\boldsymbol{v})$ (eq 16.11) and likewise for $p(\boldsymbol{v}\mid\boldsymbol{h})$ (eq 16.12) — and each individual conditional is just a sigmoid:

Together, the factorial conditionals enable block Gibbs sampling block gibbs sampling Gibbs sampling that resamples a whole conditionally-independent group at once (all h given v, then all v given h in an RBM), exploiting the model's factorial conditionals. defined in ch. 16 — open in glossary : sample all of $\boldsymbol{h}$ at once from $\boldsymbol{v}$, then all of $\boldsymbol{v}$ at once from $\boldsymbol{h}$, alternating. Step it:

Finally, because the energy is linear in the parameters, its derivatives are trivial — e.g. $\frac{\partial}{\partial W_{i,j}} E(\boldsymbol{v},\boldsymbol{h}) = -\mathrm{v}_i \mathrm{h}_j$ (eq 16.15). Efficient block Gibbs sampling and efficient derivatives are exactly what make the RBM convenient to train (chapter 18 shows how undirected models are trained from such derivatives on model samples). Training induces a representation $\boldsymbol{h}$; $\mathbb{E}_{\boldsymbol{h}\sim p(\boldsymbol{h}\mid\boldsymbol{v})} [\boldsymbol{h}]$ serves as features for $\boldsymbol{v}$.

The RBM thus distills the whole deep-learning approach to graphical models: representation learning through layers of latent variables, with efficient matrix-parametrized interactions between layers. With this language in hand, the chapters ahead describe a wide variety of deep probabilistic models.