§20.10.3–20.10.5Variational Autoencoders · Generative Adversarial Networks · Moment Matching

Two differentiable generators dominate modern generative modeling. Both pair the generator $g(\boldsymbol{z})$ with a second network — an inference net, or an adversary.

§20.10.3 Variational autoencoders

The variational autoencoder variational autoencoder A deep generative model whose encoder network performs learned approximate inference: the inference net simply defines the ELBO L, and its parameters are trained to increase L — no explicit inference targets needed. Full treatment in §20.10.3. defined in ch. 20 — open in glossary (VAE) pairs the generator (a decoder $p(\boldsymbol{x}\mid\boldsymbol{z})$) with a learned encoder $q(\boldsymbol{z}\mid\boldsymbol{x})$, and trains both by maximizing the evidence lower bound:

The whole thing is trainable by gradient descent because the reparametrization trick lets gradients flow through the sample $\boldsymbol{z}\sim q$. Training encoder and decoder together forces a smooth, predictable latent coordinate system — the VAE is an excellent manifold learner. Traverse a learned 2-D latent space (fig 20.6):

VAEs are elegant, extend to almost any architecture (DRAW, variational RNNs, importance-weighted autoencoders), and give state-of-the-art likelihoods. Their weakness: samples are often blurry — an artifact of maximizing a likelihood lower bound (which, like $D_{\mathrm{KL}}(p_\text{data}\|p_\text{model})$, spreads mass onto in-between points; fig 3.6), aggravated by the Gaussian output that tends to ignore small-but-crucial features.

§20.10.4 Generative adversarial networks

A generative adversarial network generative adversarial network (gan) A generator g(z) trained against a discriminator d(x) in a minimax game v=E_data log d + E_model log(1−d); at equilibrium samples are indistinguishable and d→½. No inference or Z-gradient needed, but training can fail to converge. defined in ch. 20 — open in glossary (GAN) replaces inference with a game. The generator $g$ makes samples; a discriminator discriminator The GAN network d(x) that estimates the probability x is a real training example rather than a generator sample; its adversary is the generator. defined in ch. 20 — open in glossary $d$ tries to tell real from fake; each fights the other:

No inference network, no partition-function gradient — just two networks in competition. Play it: the generator chases the data while the discriminator hunts the gap; at convergence the densities coincide and $d\to\tfrac12$:

That non-convergence — equilibria are saddle points of $v$, not minima of a single loss — is GANs’ central difficulty (the best-performing generator uses a non-saturating heuristic loss instead of $\min_g v$). When it works (DCGAN, LAPGAN, conditional GANs), it produces the sharpest samples of any method — a LAPGAN fooled human judges 40% of the time. The idea descends directly from noise-contrastive estimation (§18.6): “a good model should fool a classifier.” Uniquely, a GAN can place zero probability on the training points yet still trace a convincing data manifold.

§20.10.5 Generative moment matching networks

A third option drops the second network entirely. A generative moment matching network generative moment matching network A differentiable generator trained (without a discriminator) to match the moments of the data via the maximum mean discrepancy cost. defined in ch. 20 — open in glossary trains $g$ so the moments of its samples match the data’s — using the maximum mean discrepancy maximum mean discrepancy (mmd) A kernel-based cost measuring the difference in (infinitely many) moments between two distributions; zero iff the distributions are equal. Trains GMMNs. defined in ch. 20 — open in glossary (MMD), a kernel cost that compares infinitely many moments at once and is zero iff the two distributions are equal.

Next: giving these generators convolutional structure, and the fully-directed autoregressive models and autoencoder-based samplers.