Convolutional Generators · Auto-Regressive Networks · NADE · Sampling from Autoencoders

§20.10.6 Convolutional generator networks

When the generator produces images, give it convolutional structure — but run it in reverse of a recognition net. A classifier discards information going up (via pooling); a generator must add detail going down. So it uses the transpose of convolution (§9.5) to grow spatial size, and an “un-pooling” that spreads each value out to fill a larger region. These assumptions (max in a fixed corner, zeros elsewhere) are crude, but later layers learn to compensate, and the samples come out sharp with far fewer parameters than dense layers.

§20.10.7 Auto-regressive networks

A different tack drops latent variables entirely. An auto-regressive network factorizes the joint by the chain rule — every variable predicted from all the ones before it:

The chain-rule factorization

p(\boldsymbol{x}) = \prod_{i=1}^{d} p\!\left(\mathrm{x}_i \mid \mathrm{x}_{i-1}, \ldots, \mathrm{x}_1\right)

$p(\mathrm{x}_i \mid \mathrm{x}_{<i})$	each conditional, represented by a neural network — no latent variables at all	conditional
$\prod_{i=1}^{d}$	the exact joint, one factor per variable in a fixed order. The graph is COMPLETE (every variable depends on all predecessors)	exact

Generation is a left-to-right sweep; each new variable is sampled conditioned on everything already drawn:

Autoregressive generation. Each variable xᵢ is sampled from p(xᵢ | x₁…xᵢ₋₁) — conditioned on ALL the variables before it (the arcs), with the current factor highlighted in the formula. No latent variables: the joint is exactly the chain-rule product. Toggle NADE to see the weight-sharing that makes it cheap.

NADE weight sharing

An autoregressive model has no latent variables: it factorizes the joint by the chain rule and generates one variable at a time, each conditioned on all the ones before it (the arcs). A different neural net per conditional is expensive; NADE shares features to fix that.

The fully-visible belief network is the simplest form — each conditional a log-linear model (logistic/softmax regression, Frey 1998), O(d²) parameters. Neural auto-regressive networks (Bengio & Bengio 2000) replace the log-linear conditionals with neural nets and reuse hidden features across positions, escaping the tabular curse of dimensionality:

The auto-regressive family. Click a cell for detail.
	Model	Each conditional p(xᵢ\|x<ᵢ) is…
FVBN / linear	fully-visible belief net (Frey 1998)	a log-linear model (logistic/softmax regression)
Neural auto-regressive	neural AR net (Bengio & Bengio 2000)	a neural net, with hidden features REUSED across positions
NADE	neural autoregressive density estimator	a neural net with a specific weight-sharing scheme (W′_{j,k,i}=W_{k,i})

Dotted-underlined cells have explanations — click one.

The most refined member is NADE — the neural autoregressive density estimator — whose weight-sharing scheme makes one forward pass resemble a single step of RBM mean-field inference, cutting the cost to $O(n\cdot h)$ .

§20.11 Drawing samples from autoencoders

Chapter 14 showed autoencoders quietly learn the data distribution (via their tie to score matching). How to sample them? The variational autoencoder represents $p(\boldsymbol{x})$ explicitly and samples by ancestral sampling (draw $\boldsymbol{z}\sim p(\boldsymbol{z})$ , decode). Most others need a Markov chain. A denoising autoencoder defines one naturally — each step corrupts, re-encodes, decodes, and resamples:

Fig 20.11 — the Markov chain of a denoising autoencoder. From state x: (1) corrupt to x̃ ∼ C(x̃|x), (2) encode h = f(x̃), (3) decode to the reconstruction distribution’s parameters ω = g(h), (4) sample the next state x from p(x|ω). Its stationary distribution consistently estimates p_data.

The chain works because the autoencoder has learned a vector field pointing back toward the data manifold (§14.5.1) — recall it:

The denoising autoencoder's learned reconstruction field (from §14.5.1). Each Markov-chain step nudges the state along these arrows toward the data manifold, then adds noise — so the walk explores the manifold and its stationary distribution estimates p_data. A contractive autoencoder's tangent estimate induces a similar manifold-diffusion walk.

corruption width σ = 26

Each arrow is the optimal denoising reconstruction minus the input, g(f(x̃)) − x̃ = 𝔼[x | x̃] − x̃ — the center of mass of the clean points that could have produced x̃, minus x̃. It estimates the score ∇ₓ log p(x) (up to a factor σ²): arrows climb toward the manifold and shrink to dots where the estimated density is flat or maximal. Moderate σ: a smooth field pointing cleanly back to the spiral — zeros ON the manifold (density maxima) and between the arms (density minima), long arrows in low-probability gaps where a step uphill gains the most.

Next: making that autoencoder Markov chain a first-class generative model — generative stochastic networks and other schemes.

Check yourself — convolutional generators, autoregressive nets & autoencoder sampling

1.Why does a convolutional GENERATOR use transpose convolution and un-pooling, opposite to a recognition network?

2.How does an autoregressive network define its distribution, and what latent variables does it use?

3.What makes NADE more efficient than having a separate neural network for each conditional?

4.How are samples drawn from a variational autoencoder versus a denoising autoencoder?

5.Predict the outcome — in AutoRegLab you click 'generate next' to fill x₄. What does the widget show about its conditioning?

5 questions