Variational Inference with Discrete Latent Variables

§19.4.1 Discrete latent variables

Mean field is at its simplest when the latents are discrete. For binary $\boldsymbol{h}$ , a factorial $q$ is just a vector $\hat{\boldsymbol{h}}$ of probabilities, $q(h_i = 1 \mid \boldsymbol{v}) = \hat{h}_i$ . We optimize those probabilities to maximize $\mathcal{L}$ — and because this optimization sits in the inner loop of learning, it must be fast. Rather than gradient descent, we iterate fixed-point equations : solve $\partial \mathcal{L} / \partial \hat{h}_i = 0$ for one $\hat{h}_i$ at a time (eq 19.18) and cycle until convergence.

The running example is binary sparse coding. An input $\boldsymbol{v}$ is the sum of components, each switched on/off by a hidden unit, plus Gaussian noise:

The binary sparse coding model (eqs 19.19–19.20)

p(h_i = 1) = \sigma(b_i), \qquad p(\boldsymbol{v} \mid \boldsymbol{h}) = \mathcal{N}(\boldsymbol{v};\ \boldsymbol{W}\boldsymbol{h},\ \boldsymbol{\beta}^{-1})

$p(h_i = 1) = \sigma(b_i)$	each component is present with a prior probability set by a learnable bias bᵢ	Bernoulli
$\boldsymbol{W}\boldsymbol{h}$	the active components sum into a reconstruction — a linear decoder	n
$\boldsymbol{\beta}^{-1}$	learnable diagonal noise precision. Together these make p(h\|v) intractable	n × n

Why the posterior is intractable. Every pair $h_i, h_j$ are co-parents of the same visibles, so observing $\boldsymbol{v}$ activates a path between every pair of hidden units — the posterior $p(\boldsymbol{h}\mid\boldsymbol{v})$ is a complete graph over $\boldsymbol{h}$ (fig 19.2). No variable-elimination shortcut helps; the maximum-likelihood gradient’s expectation over $p(\boldsymbol{h}\mid\boldsymbol{v})$ is brute-force expensive.

Fig 19.2 — the generative model p(h,v) (left) is sparse: four hidden causes point down to the visibles. But the posterior p(h|v) (right) is the COMPLETE graph over the hidden units — observing v couples every pair of co-parents by explaining away.

Mean field to the rescue. Assume $q(\boldsymbol{h}\mid\boldsymbol{v}) = \prod_i q(h_i\mid\boldsymbol{v})$ (eq 19.28). The ELBO (eqs 19.29–19.36) becomes a handful of arithmetic operations, and solving $\partial\mathcal{L}/\partial \hat{h}_i = 0$ gives a closed-form update for each unit:

The mean-field fixed-point update (eq 19.44)

\hat{h}_i = \sigma\!\left(b_i + \boldsymbol{v}^\top \boldsymbol{\beta} \boldsymbol{W}_{:,i} - \tfrac{1}{2}\boldsymbol{W}_{:,i}^\top \boldsymbol{\beta} \boldsymbol{W}_{:,i} - \sum_{j\neq i} \boldsymbol{W}_{:,j}^\top \boldsymbol{\beta} \boldsymbol{W}_{:,i}\, \hat{h}_j\right)

$\boldsymbol{v}^\top \boldsymbol{\beta} \boldsymbol{W}_{:,i}$	the fixed input drive: how well unit i's weights match the input. Constant during inference	scalar
$- \sum_{j\neq i} \boldsymbol{W}_{:,j}^\top \boldsymbol{\beta} \boldsymbol{W}_{:,i}\, \hat{h}_j$	INHIBITION: unit i is suppressed by every other active unit j, in proportion to their weight overlap. This is the recurrent, changing message	scalar
$\sigma(\cdot)$	squashes to a probability. Iterating this for i = 1…m IS a recurrent inference network	probability

The inhibition term is the whole story: two units whose weight vectors align inhibit each other, so only the one that best explains the input survives. Iterate the competition:

The mean-field fixed point as a recurrent inference network. Two hidden units explain one input; raise their weight alignment and each inhibits the other. Cyclic updates (ĥ ← σ(drive − align·ĥ')) then drive one unit to 1 and the other to 0 — the mean-field attempt at explaining away. The true posterior is bimodal (either unit could be the cause) but the factorial q is forced to pick one mode.

ĥ₁

0.60

ĥ₂

0.40

drive d = 2.0align a = 5.0

sweeps: 0

still co-active (undecided)

High alignment (overlapping weights) makes the two units compete: each inhibits the other, so cyclic updates drive one to 1 and the other to 0 — the mean-field attempt at explaining away. The true posterior is bimodal (either unit could explain v), but a factorial q can only capture one mode. Drop alignment and both stay active.

Sparse coding as an iterative autoencoder

Rewriting the update (eq 19.45), unit $i$ ‘s drive becomes $\boldsymbol{v} - \sum_{j\neq i}\boldsymbol{W}_{:,j}\hat{h}_j$ — the residual left over once the other units’ contributions are subtracted. So each unit tries to encode what the others missed: sparse coding runs as an iterative autoencoder, repeatedly encoding and decoding to fix the reconstruction. But because the factorial $q$ cannot be bimodal, the explaining-away competition (which really produces a multimodal posterior) is collapsed onto a single mode — the same reverse-KL behavior as fig 3.6.

One update touches a single unit. Some models (deep Boltzmann machines) let you solve for many $\hat{h}_i$ at once; binary sparse coding does not, so we fall back on damping — solve every unit’s optimal value, then move them all a small step that way. It no longer guarantees an increase in $\mathcal{L}$ each step, but works well in practice. Next: the continuous case and the tool behind it — calculus of variations, and learned inference.

Check yourself — discrete mean-field inference

1.Why is the posterior p(h|v) of the binary sparse coding model intractable, even though the generative model p(h,v) is sparse?

2.Why are mean-field parameters computed with fixed-point equations rather than gradient descent?

3.In the mean-field update ĥᵢ = σ(drive − Σⱼ≠ᵢ (weight overlap)·ĥⱼ), what does the sum term do?

4.The true posterior of two aligned competing units is bimodal (either unit could explain v). What does the factorial mean-field q do?

5.Predict the outcome — in MeanFieldLab you set a high weight alignment and click 'iterate' several times. What happens to ĥ₁ and ĥ₂?

5 questions