Boltzmann Machines · Restricted Boltzmann Machines

Part III’s tools now build actual generative models. We start where the deep learning story began: Boltzmann machines.

§20.1 Boltzmann machines

A Boltzmann machine is an energy-based model over binary variables. With hidden units it splits into visible $\boldsymbol{v}$ and latent $\boldsymbol{h}$ :

The Boltzmann machine energy (eqs 20.2–20.3)

E(\boldsymbol{v}, \boldsymbol{h}) = -\boldsymbol{v}^\top \boldsymbol{R}\boldsymbol{v} - \boldsymbol{v}^\top \boldsymbol{W}\boldsymbol{h} - \boldsymbol{h}^\top \boldsymbol{S}\boldsymbol{h} - \boldsymbol{b}^\top\boldsymbol{v} - \boldsymbol{c}^\top\boldsymbol{h}

$\boldsymbol{R}, \boldsymbol{S}$	visible–visible and hidden–hidden couplings — the FULLY connected Boltzmann machine allows them (the RBM will forbid both)	weight matrices
$\boldsymbol{W}$	the visible–hidden couplings	n_v × n_h
$E$	low energy = high probability, via P(v,h)=exp(−E)/Z. Adding hidden units makes the BM a UNIVERSAL approximator of discrete distributions	scalar

Like all Boltzmann machines it has an intractable partition function, so training uses the maximum-likelihood approximations of chapter 18. One striking property: the update for a weight between two units depends only on the two units’ co-statistics — a local rule.

Fire together, wire together

In the positive phase, two units that frequently activate together have their connection strengthened — a Hebbian learning rule, and one of the oldest hypotheses for how biological neurons learn. It is “biologically plausible” in a way back-propagation is not: a synapse can update from only the firing of the two cells it physically touches, with no separate network to carry gradients backward. (The negative phase is harder to explain biologically — one speculation from §18.2 is that dream sleep supplies its samples.)

§20.2 Restricted Boltzmann machines

The restricted Boltzmann machine (harmonium) is the workhorse — the building block stacked into the deeper models below. “Restricted” = no visible–visible and no hidden–hidden edges (drop $\boldsymbol{R}$ and $\boldsymbol{S}$ ), leaving a bipartite graph:

Fig 20.1 — models built from RBMs. (a) The RBM: a bipartite undirected graph, visible below, hidden above, no intra-layer edges. (b) A deep belief network: undirected top pair, directed below. (c) A deep Boltzmann machine: fully undirected, several hidden layers. (b) and (c) are developed in §20.3–20.4.

Its energy is the interaction term alone, $E(\boldsymbol{v},\boldsymbol{h}) = -\boldsymbol{b}^\top\boldsymbol{v} - \boldsymbol{c}^\top\boldsymbol{h} - \boldsymbol{v}^\top\boldsymbol{W}\boldsymbol{h}$ (eq 20.5). The partition function is provably intractable, so $P(\boldsymbol{v})$ can’t be evaluated — but the bipartite restriction buys everything that matters.

§20.2.1 The gift of the bipartite structure

Because no unit shares a layer with a unit it depends on, the layer-wise conditionals factorize, and each is a plain sigmoid:

Factorial, sigmoidal conditionals (eq 20.14)

P(\mathrm{h}_j = 1 \mid \boldsymbol{v}) = \sigma\!\left(c_j + \boldsymbol{v}^\top \boldsymbol{W}_{:,j}\right), \qquad P(\boldsymbol{h}\mid\boldsymbol{v}) = \prod_j P(\mathrm{h}_j \mid \boldsymbol{v})

$\prod_j$	the hidden units are conditionally INDEPENDENT given v (and vice versa), so the whole layer is one product of Bernoullis	factorial
$\sigma(c_j + \boldsymbol{v}^\top\boldsymbol{W}_{:,j})$	each conditional is exactly the RBM sigmoid of §16.7.1 — cheap to evaluate AND to sample from	probability

Factorial conditionals mean block Gibbs sampling: draw the entire hidden layer at once, then the entire visible layer, alternating. Replay it — this is the Markov chain the RBM is trained and sampled with:

Block Gibbs sampling in an RBM (from §16.7.1). The bipartite restriction makes P(h|v) and P(v|h) factorial, so each "sample" updates a whole layer at once from its sigmoid conditionals. This efficient sampling is exactly what makes the RBM trainable despite its intractable partition function.

step 0 · v = [1, 0, 1] · h = [0, 0]

next: sample the HIDDEN block from P(hᵢ=1 | v)

No hidden–hidden or visible–visible edges means every unit in a layer is conditionally independent of the rest of its layer given the other layer — so the whole block samples at once, each unit from a simple sigmoid. Alternating h, v, h, v… is block Gibbs sampling, and it is what makes the RBM cheap to train (ch18).

§20.2.2 Training the RBM

Because we can evaluate and differentiate $\tilde{P}(\boldsymbol{v})$ and sample efficiently by block Gibbs, the RBM trains with any of the intractable-Z methods of chapter 18:

Why the RBM is the easy undirected model to train. Click a cell for detail.
	What training needs	The RBM gives it
A tractable positive phase	the posterior p(h\|v)	exact, in closed form
A cheap negative phase	samples from the model	efficient block Gibbs
Compatible training methods	an intractable-Z method	CD · SML · ratio matching

Dotted-underlined cells have explanations — click one.

This combination — exact posterior, easy sampling — is what makes the RBM the reusable brick of deep probabilistic models. Stack it and the difficulties return: the deep belief networks and deep Boltzmann machines next combine intractable partition functions and intractable inference.

Check yourself — Boltzmann machines & RBMs

1.What does adding hidden units to a Boltzmann machine achieve?

2.Why is Boltzmann machine learning called 'local' and biologically plausible?

3.What does the 'restricted' in restricted Boltzmann machine mean, and why does it matter?

4.Why is the RBM the easiest undirected model to train, compared to a deep Boltzmann machine?

5.Predict the outcome — in BlockGibbsLab you click 'sample the hidden block' for an RBM. What is computed?

5 questions