DAE Markov Chains · Walk-Back · Generative Stochastic Networks · Other Schemes

§20.11.1 The chain estimates the data distribution

The denoising-autoencoder Markov chain is not just a sampling trick — it is a consistent estimator. If the autoencoder’s reconstruction distribution $p(\boldsymbol{x}\mid\tilde{\boldsymbol{x}})$ converges to the true clean-given-corrupt conditional, then the chain’s stationary distribution consistently estimates $p_\text{data}$ (Bengio et al. 2014) — an implicit model, defined by the process rather than by a formula.

§20.11.2 Clamping. Like a Boltzmann machine, the chain can sample a conditional $p(\boldsymbol{x}_f\mid\boldsymbol{x}_o)$ : hold the observed units $\boldsymbol{x}_o$ fixed and resample only the free units $\boldsymbol{x}_f$ each step (needs the transition operator to satisfy detailed balance). This is how a denoising autoencoder fills in the missing half of an image.

§20.11.3 Walk-back training

A one-step denoising objective sees only configurations near the data. The walk-back procedure runs several stochastic encode–decode steps from a training example (as contrastive divergence does) and penalizes the reconstructions along the way — reaching farther, so it eliminates spurious modes more efficiently:

Walk-back training (§20.11.3). Instead of one denoising step, take several from a data point, penalizing each reconstruction — pushing down spurious modes that lie far from the data.

11 · Start at a training example

Initialize the chain at a real data point x⁽⁰⁾ (as contrastive divergence does), where the model already has support.

step 1 / 3

§20.12 Generative stochastic networks

A generative stochastic network (GSN) generalizes the DAE chain by adding latent variables $\boldsymbol{h}$ to it. Rather than specifying a joint distribution, a GSN parametrizes the generative process — two conditionals that define one step:

The two conditionals of a GSN step

p\!\left(\boldsymbol{x}^{(k)} \mid \boldsymbol{h}^{(k)}\right), \qquad p\!\left(\boldsymbol{h}^{(k)} \mid \boldsymbol{h}^{(k-1)}, \boldsymbol{x}^{(k-1)}\right)

$p(\boldsymbol{x}^{(k)}\mid\boldsymbol{h}^{(k)})$	the "reconstruction distribution" — how to emit the next visible from the current latent (as in a DAE, RBM, DBN, DBM)	conditional
$p(\boldsymbol{h}^{(k)}\mid\boldsymbol{h}^{(k-1)}, \boldsymbol{x}^{(k-1)})$	how to update the LATENT state from the previous latent and visible — the extra piece GSNs add over a plain DAE chain	conditional

The joint is defined only implicitly, as the chain’s stationary distribution (existence needs the mild mixing conditions of §17.3). GSNs are trained by reconstruction log-probability (clamping $\boldsymbol{x}^{(0)}$ to data, via the reparametrization trick and walk-back). Discriminant GSNs (§20.12.1) back-propagate reconstruction only over output variables to model $p(\boldsymbol{y}\mid\boldsymbol{x})$ — structured output learning.

§20.13 Other generation schemes

Most models here sample by MCMC or ancestral sampling, but the space is open:

Two more generation schemes. Click a cell for detail.
	Scheme	How it generates
Diffusion inversion	diffusion inversion (Sohl-Dickstein 2015)	learn to REVERSE a diffusion that gradually adds entropy — train a transition operator to undo one noising step at a time
Approximate Bayesian computation	ABC (Rubin 1984)	reject or modify samples so their selected moments match the target

Dotted-underlined cells have explanations — click one.

A seed of what came next

Diffusion inversion (2015) looked like a minor entry in 2016. It was the forerunner of the diffusion models that, a few years after this book, would overtake GANs on image synthesis — the same core idea of learning to reverse a step-by-step noising process, scaled up. It is a fitting reminder that the generative-model landscape this chapter surveys never stopped moving.

Only one question remains for these models — how do we know one is better than another? Next: evaluating generative models, and the book’s conclusion.

Check yourself — GSNs & other generation schemes

1.In what sense does the denoising-autoencoder Markov chain estimate the data distribution?

2.How does a DAE Markov chain sample a conditional distribution p(x_f | x_o)?

3.What problem does walk-back training address, and how?

4.How does a generative stochastic network (GSN) differ from a denoising autoencoder, and how is its distribution defined?

5.What is diffusion inversion, and why is it historically notable?

5 questions