DBM Parameter Learning · Layer-Wise Pretraining · Jointly Training DBMs

§20.4.3 Learning: two intractable problems at once

Training a DBM is the hardest case in the book: it must confront both an intractable partition function and an intractable posterior. The variational lower bound still contains $\log Z$ :

The DBM variational lower bound (eq 20.35)

\mathcal{L}(Q, \boldsymbol{\theta}) = \sum_{i,j} v_i W^{(1)}_{i,j}\hat{h}^{(1)}_j + \sum_{j,k} \hat{h}^{(1)}_j W^{(2)}_{j,k}\hat{h}^{(2)}_k - \log Z(\boldsymbol{\theta}) + H(Q)

$\text{the two sums}$	the expected energy under the mean-field Q — the tractable, positive-phase part built from the ĥ of §20.4.2	scalar
$-\log Z(\boldsymbol{\theta})$	the still-intractable partition function → needs the negative-phase / SML machinery of chapter 18	scalar
$H(Q)$	the entropy of the factorial mean-field approximation — cheap	scalar

So DBMs are trained with variational SML: run mean-field inference (§19) for the positive phase, persistent Gibbs chains (§18) for the negative phase.

Variational stochastic maximum likelihood for a DBM (Algorithm 20.1)

1initialize persistent negative chains $\tilde{V}, \tilde{H}^{(1)}, \tilde{H}^{(2)}$
2for each minibatch:
3positive phase: run mean-field to convergence to get $\hat{H}^{(1)}, \hat{H}^{(2)}$ ⓘ
4negative phase: run $k$ two-block Gibbs steps on the persistent chainsⓘ
5update: $\Delta W \leftarrow \frac{1}{m}(\text{data} \cdot \hat{H}) - \frac{1}{m}(\tilde{V}\cdot\tilde{H})$ ⓘ

Other chapter-18 methods don’t apply: contrastive divergence is slow (a DBM can’t sample hidden units given visibles in closed form, so each CD step would need a fresh burn-in), and pseudolikelihood/ratio matching need the unnormalized probability, which the variational bound only lower-bounds.

§20.4.4 Layer-wise pretraining

Training a DBM by SML from a random initialization usually fails — it either learns poorly or collapses to something no better than a single RBM. The classic fix is the same greedy pretraining that built the DBN, with a twist:

Greedy pretraining then joint training of a DBM (fig 20.4). Each layer is first an RBM; the weights are adjusted before being combined.

11 · Pretrain each layer as an RBM

Train the first RBM on the data, the second on the first's hidden samples, and so on — exactly as for a DBN. Each learns a good initial layer in isolation.

step 1 / 3

State-of-the-art results also use a bit of partial mean field in the negative phase (one mean-field step before the Gibbs sampling).

§20.4.5 Jointly training deep Boltzmann machines

Greedy pretraining is awkward: you can’t monitor the full DBM while training the first RBM, the software needs many components, and a separate MLP classifier on top loses the Boltzmann machine’s ability to reason about missing inputs. Two routes train a DBM jointly from scratch:

Two ways to jointly train a DBM. Click a cell for detail.
	Method	How it enables joint training	Result
Centered DBM	centered DBM	subtract a fixed µ from all states so x−µ ≈ 0 at init → better-conditioned Hessian	good likelihood & samples; weak classifier
MP-DBM	multi-prediction DBM	treat the mean-field equations as recurrent nets and back-propagate through the inference graph	strong classifier & missing-input reasoning; defective p(v)

Dotted-underlined cells have explanations — click one.

MP-DBM: train the network you actually run

The MP-DBM ’s key move is to stop treating inference as a fixed procedure and start treating it as part of the model to be trained. Unrolling the mean-field fixed point gives a recurrent network for each possible “predict these variables from those” problem; back-propagating through it trains inference to be accurate as it is really used. It even echoes dropout: both share parameters across many computational graphs — dropout across which units are present, the MP-DBM across which inputs are observed (unobserved ones become latents to infer, not deleted). The centered DBM takes the other route — leave the objective alone, fix the optimization landscape.

Next: extending Boltzmann machines beyond binary data to real values — Gaussian, covariance, and spike-and-slab RBMs.

Check yourself — DBM training

1.Why is training a DBM harder than training an RBM?

2.In variational SML for a DBM, what plays the positive-phase and negative-phase roles?

3.Why must RBM weights be adjusted (e.g. halved) before combining them into a DBM, unlike for a DBN?

4.How does the centered DBM make joint training (without greedy pretraining) succeed?

5.What is the MP-DBM's core idea, and its main tradeoff?

5 questions