§20.6–20.9Convolutional & Sequential Boltzmann Machines · Back-Prop through Random Operations

§20.6 Convolutional Boltzmann machines

Images strain any dense model, so — as in chapter 9 — replace matrix multiplication with convolution. Convolutional RBMs work well, but pooling is awkward in an energy-based model: a hard “pooling unit = $\max_i d_i$” over $n$ detectors would cost $2^n$ energy evaluations to normalize. Probabilistic max pooling probabilistic max pooling A way to include max-pooling in an energy-based (convolutional Boltzmann) model by constraining at most one detector unit per region to be active. defined in ch. 20 — open in glossary (Lee et al. 2009) fixes this by constraining at most one detector active per region — so there are only $n+1$ states (each detector on, or all off), and the pooling unit is on iff some detector is. It’s efficient, but forces the detectors to be mutually exclusive and forbids overlapping pooling regions, which limits accuracy versus supervised convnets.

§20.7 Structured and sequential outputs

A Boltzmann machine can be made conditional — model $p(\boldsymbol{y}\mid \boldsymbol{x})$ where the outputs $\boldsymbol{y}$ obey constraints (e.g. a coherent speech waveform), or a sequence $p(x^{(t)}\mid x^{(t-1)},\ldots)$. A conditional RBM (Taylor et al. 2007) makes the RBM’s biases a linear function of the past $m$ frames; different histories switch different hidden units on, reshaping the distribution over the next frame. The RNN-RBM rnn-rbm A sequence model where an RNN emits all of an RBM's parameters at each time step; trained by back-propagating a CD gradient through time. defined in ch. 20 — open in glossary (Boulanger-Lewandowski et al. 2012) goes further: an RNN emits all of the RBM’s parameters (weights included) at each step — trained by back-propagating a contrastive-divergence gradient through time.

§20.8 Other Boltzmann machines

The framework is a rich, still-open space: discriminative RBMs maximize $\log p(y\mid v)$ (best combined with the generative criterion); higher-order Boltzmann machines use energy terms that multiply more than two variables (three-way units can model video transformations, or gate features by a class label). Designing a new one is harder than adding a neural-net layer — you must find an energy function that keeps all the needed conditionals tractable.

§20.9 Back-propagation through random operations

Generative nets need to sample, but a sampling step seems to block gradients. The reparametrization trick reparametrization trick Rewrite a sample y∼N(µ,σ²) as y=µ+σz with z∼N(0,1) external noise, turning sampling into a differentiable function so gradients flow through it (stochastic back-propagation). Continuous variables only. defined in ch. 20 — open in glossary dissolves the problem: pull the randomness out into an external input.

Direct sampling from $\mathcal{N}(\mu,\sigma^2)$ is a stochastic node with no differentiable path to its parameters; rewriting it as $y=\mu+\sigma z$ makes it trainable. Toggle between the two views:

More generally, any $y \sim p(y\mid\omega)$ can be rewritten $y = f(z;\omega)$ (eq 20.57) with $z$ external — trainable as long as $f$ is continuous and differentiable, which requires $y$ continuous. For discrete $y$ the trick fails (a step function has zero gradient), and we need the REINFORCE estimator instead — next.