Deep Belief Networks · Deep Boltzmann Machines · Mean-Field Inference

Stack RBMs and the tractability of §20.2 is lost — but two famous deep models result.

§20.3 Deep belief networks

A deep belief network (DBN) is a hybrid: the top two layers form an undirected RBM, while every lower pair is directed downward. It is trained greedily, layer by layer — the recipe that launched the 2006 deep-learning renaissance:

Greedy layer-wise training of a DBN (§20.3). Each layer is trained as an RBM on the samples of the layer below, then frozen.

11 · Train the first RBM on the data

Train an RBM to maximize E_{v∼data} log p(v) with contrastive divergence or SML. Its weights become the DBN's first (lowest) layer.

step 1 / 3

DBNs inherit the worst of both worlds: inference is intractable (explaining away in the directed layers and the undirected top pair), and so is the partition function. Its log-likelihood must be estimated with AIS. In practice the trained DBN’s weights were mainly used to initialize an MLP (eqs 20.22–23) for discriminative fine-tuning — a heuristic upward pass that ignores the model’s top-down and lateral interactions. DBNs are now largely historical, but they proved deep architectures could be trained at all.

§20.4 Deep Boltzmann machines

A deep Boltzmann machine (DBM) is entirely undirected with several hidden layers and no intra-layer edges:

The DBM energy (eq 20.25, three hidden layers)

E(\boldsymbol{v}, \boldsymbol{h}^{(1)}, \boldsymbol{h}^{(2)}, \boldsymbol{h}^{(3)}) = -\boldsymbol{v}^\top \boldsymbol{W}^{(1)}\boldsymbol{h}^{(1)} - \boldsymbol{h}^{(1)\top}\boldsymbol{W}^{(2)}\boldsymbol{h}^{(2)} - \boldsymbol{h}^{(2)\top}\boldsymbol{W}^{(3)}\boldsymbol{h}^{(3)}

$\boldsymbol{W}^{(2)}, \boldsymbol{W}^{(3)}$	hidden–hidden couplings between LAYERS — unlike an RBM, deeper layers interact (but never within a layer)	weight matrices
$\text{only neighbor-layer terms}$	each layer connects only to its neighbors → the graph is bipartite (odd layers vs even layers), just like a big RBM	bipartite

Because the layers split into an even group (visible + even hidden) and an odd group, the DBM is bipartite — so conditioning on one group makes the other factorial, and Gibbs sampling needs just two block updates (all even, then all odd), no matter how many layers. The conditionals are the familiar RBM sigmoids, now with input from both neighboring layers (eqs 20.26–28).

§20.4.1 Why the DBM’s harder-looking model is easier to infer in

Counterintuitively, the DBM’s posterior is simpler than the DBN’s, and that simplicity permits a better approximation:

DBN vs DBM. Click a cell for detail.
	Aspect	Deep belief network	Deep Boltzmann machine
Directedness	graph type	hybrid: directed below, undirected top	fully undirected
Inference	approximating the posterior	heuristic MLP upward pass	proper mean field (optimized)
Top-down feedback	does inference use it?	no (upward MLP only)	yes
Training	how it is trained	greedy layer-wise, then MLP fine-tune	variational SML (both hard problems)

Dotted-underlined cells have explanations — click one.

§20.4.2 DBM mean-field inference

The DBM posterior over all hidden layers doesn’t factorize (the inter-layer weights couple them). So we approximate it with a fully factorial mean field $Q$ :

The mean-field fixed-point updates (eqs 20.29, 20.33–20.34)

Q(\boldsymbol{h}^{(1)}, \boldsymbol{h}^{(2)} \mid \boldsymbol{v}) = \prod_j Q(h_j^{(1)}\mid\boldsymbol{v})\prod_k Q(h_k^{(2)}\mid\boldsymbol{v}), \quad \hat{h}_j^{(1)} = \sigma\!\Big(\sum_i v_i W_{i,j}^{(1)} + \sum_k W_{j,k}^{(2)} \hat{h}_k^{(2)}\Big)

$\prod_j \cdots \prod_k$	the factorial (mean-field) assumption: every hidden unit independent given v — parametrized by its Bernoulli mean ĥ	factorial Q
$\sum_i v_i W^{(1)}_{i,j}$	bottom-up drive from the layer below (the visibles)	scalar
$\sum_k W^{(2)}_{j,k}\hat{h}^{(2)}_k$	TOP-DOWN feedback from the layer above — what the DBN's heuristic MLP omits. Iterate ĥ⁽¹⁾, ĥ⁽²⁾ to a fixed point → the optimized posterior	scalar

Alternating $\hat{h}^{(1)} = \sigma(\ldots)$ and $\hat{h}^{(2)} = \sigma(\ldots)$ to a fixed point maximizes the variational bound — the same fixed-point mean field as §19.4.1, now with bidirectional messages. Ten iterations suffice on MNIST for a learning gradient. Replay the competition dynamics that drive it:

Mean-field fixed-point iteration (from §19.4.1), the engine of DBM inference. In a DBM each unit's update mixes bottom-up drive with top-down feedback; here two units compete to explain the input, and iterating the σ(·) update settles them to a fixed point that (approximately) optimizes the variational bound — exactly the ĥ⁽¹⁾/ĥ⁽²⁾ alternation of eqs 20.33–34.

ĥ₁

0.60

ĥ₂

0.40

drive d = 2.0align a = 5.0

sweeps: 0

still co-active (undecided)

High alignment (overlapping weights) makes the two units compete: each inhibits the other, so cyclic updates drive one to 1 and the other to 0 — the mean-field attempt at explaining away. The true posterior is bimodal (either unit could explain v), but a factorial q can only capture one mode. Drop alignment and both stay active.

Next: how a DBM is actually trained — confronting both intractable quantities at once — and how to get it to train at all.

Check yourself — DBNs & DBMs

1.What is the structure of a deep belief network, and how is it trained?

2.Why can a deep Boltzmann machine be Gibbs-sampled in just two block updates regardless of its depth?

3.Counterintuitively, why is inference in a DBM better than in a DBN, despite the DBM looking more complex?

4.In the DBM mean-field update ĥ⁽¹⁾ⱼ = σ(Σᵢ vᵢW⁽¹⁾ᵢⱼ + Σₖ W⁽²⁾ⱼₖ ĥ⁽²⁾ₖ), what does the second sum contribute?

5.Predict the outcome — in MeanFieldLab (the DBM inference engine) you raise the alignment and iterate. What happens, and what does it illustrate about DBM inference?

5 questions