Boltzmann Machines for Real-Valued Data · Deep Learning Interactive

§20.5 Beyond binary data

Boltzmann machines were born binary, but images and audio live in the reals. Treating a grayscale pixel’s intensity as the probability of a binary pixel works for evaluation but produces noisy samples. Better: build Boltzmann machines that define a genuine density over real values.

§20.5.1 Gaussian-Bernoulli RBMs

The most common real-valued RBM — the Gaussian-Bernoulli RBM — keeps binary hidden units but makes the visible conditional a Gaussian whose mean is set by the hidden units:

The Gaussian-Bernoulli visible conditional (eq 20.38)

p(\boldsymbol{v} \mid \boldsymbol{h}) = \mathcal{N}\!\left(\boldsymbol{v};\ \boldsymbol{W}\boldsymbol{h},\ \boldsymbol{\beta}^{-1}\right)

$\boldsymbol{W}\boldsymbol{h}$	the conditional MEAN — the active hidden units add up their weight columns to place the Gaussian's center	n_v
$\boldsymbol{\beta}^{-1}$	the covariance, a FIXED (usually diagonal) precision — it does NOT depend on h. So the model captures only the conditional mean, never the covariance	n_v × n_v

The energy just adds the terms of that log-Gaussian involving $\boldsymbol{v}$ (eq 20.42). Note the hidden units move the Gaussian’s center but leave its shape fixed — see a 1-D slice of $p(v\mid h)$ , whose mean shifts with the code but whose spread is constant:

The Gaussian conditional p(vᵢ|h) of a Gaussian-Bernoulli RBM (reused explorer). The mean is (Wh)ᵢ — set by which hidden units are active — but the variance β⁻¹ is fixed. Slide μ to see the center move; the width never responds to the hidden code. That fixed spread is exactly the model's blind spot.

μ0.00σ1.00

peak at x = μ = 0.0; inflection points at μ ± σ; precision β = 1/σ² = 1.00

§20.5.2 Modeling conditional covariance

For natural images, most of the information lives in the covariance between pixels — the relationships, not the absolute intensities. A Gaussian RBM, modeling only the conditional mean, is blind to this. Three richer models let the hidden units shape the covariance:

Mean-only vs conditional-covariance models. Left: a Gaussian RBM’s conditional is an isotropic blob whose CENTER the hidden units move. Right: a covariance model lets the hidden units reshape and re-orient the covariance ellipse — capturing pixel relationships.

Three undirected models of conditional covariance. Click a cell for detail.
	Model	How it models covariance	Training cost
mcRBM	mean & covariance RBM	separate MEAN units and COVARIANCE units; covariance units constrain the precision along learned directions r⁽ʲ⁾	expensive — needs HMC
mPoT	mean-product of Student-t	Gaussian mean units + Student-t covariance units (non-diagonal Gaussian conditional on x)	expensive — needs HMC
ssRBM	spike-and-slab RBM	binary SPIKE hᵢ gates a real-valued SLAB sᵢ; active spikes add variance along W:,ᵢ, pinching the precision matrix	cheap — CD/PCD with Gibbs

Dotted-underlined cells have explanations — click one.

Spike and slab, in one sentence

The ssRBM gives each feature two switches: a binary spike $h_i$ deciding whether component $\boldsymbol{W}_{:,i}$ is present, and a real slab $s_i$ setting how strongly — and an active spike adds variance along that component’s direction, so the hidden units literally sculpt the conditional covariance. Because this needs neither matrix inversion (unlike the mcRBM ) nor Hamiltonian Monte Carlo, it trains with ordinary Gibbs-based CD/PCD — which is why convolutional ssRBMs produce excellent natural-image samples.

Next: giving Boltzmann machines spatial structure and sequence memory — convolutional and sequential models, and back-prop through random operations.

Check yourself — real-valued Boltzmann machines

1.What does a Gaussian-Bernoulli RBM model, and what is its key limitation?

2.Why is a Gaussian RBM a poor fit for natural images specifically?

3.How does the spike-and-slab RBM (ssRBM) model conditional covariance?

4.What is the ssRBM's main advantage over the mcRBM and mPoT models?

5.Predict the outcome — in the reused DistributionExplorer showing a Gaussian-Bernoulli conditional p(vᵢ|h), you change which hidden units are active (moving the mean). What happens to the curve's width?

5 questions