Evaluating Generative Models · Conclusion · Deep Learning Interactive

§20.14 Evaluating generative models

Researchers building generative models constantly need to answer one question: is model A better than model B? — usually to show that a newly invented model captures some distribution better than what came before. It sounds routine. It is one of the subtlest tasks in the field, because the number a benchmark reports often does not measure the thing we actually care about.

The pitfalls below are not edge cases; they are the default way a naïve comparison goes wrong. Click any cell for the reasoning.

Six ways a generative-model comparison misleads (§20.14). Click a cell.
	Pitfall	What goes wrong
Bounds vs estimates	a bound and an estimate are not comparable	a lower BOUND on model B beats a stochastic ESTIMATE for model A — which is better?
AIS mis-estimates Z	a bad partition-function estimate looks like a good model	use AIS to estimate log Z, then report log p̃(x) − log Z
Preprocessing	any change to the input changes the task	even tiny preprocessing differences are completely unacceptable
Real vs binary	likelihoods must live on the same space	MNIST as a real vector, as binary, or as Bernoulli probabilities — pick one
Good samples ≠ good model	realistic samples do not certify a good density	a very poor probabilistic model can produce very good-looking samples
Likelihood is gameable	high likelihood can measure nothing useful	real-valued MNIST models win by shrinking variance on constant background pixels

Dotted-underlined cells have explanations — click one.

The metric must match the intended use

Theis et al. (2015) drive the point home: there is no single right metric, because there is no single use of a generative model. Some models should assign high probability to most realistic points; others should rarely assign high probability to unrealistic points. Those two goals are exactly the two directions of the KL divergence from the very start of the book (fig 3.6) — and the choice between them is a modeling decision, not a detail:

Fit a single Gaussian q to a bimodal target p. Forward D(p_data‖p_model) (maximum likelihood) is mode-COVERING — q spreads to span every mode, so it never misses a realistic point but places mass in the empty valley (unrealistic points). Reverse D(p_model‖p_data) (variational) is mode-SEEKING — q locks onto one mode, rarely emitting unrealistic points but ignoring the other. Neither is "correct" — the right one depends on what you need the model to do.

μ = 0.00σ = 1.20

D(q‖p) = 1.687

Reverse KL (variational inference) is MODE-SEEKING: putting q-mass in the empty valley is heavily penalized, so the best q locks onto ONE mode and ignores the other.

Even when each metric is restricted to the task it suits best, all metrics in current use have serious weaknesses. So one of the most important open problems in generative modeling is not only how to build better models, but how to measure progress at all — designing new evaluation techniques is itself a research frontier.

§20.15 Conclusion

Training generative models with hidden units is a powerful way to make a model understand the world represented in its training data. The payoff is that a single trained model gives you two things at once:

What a latent-variable generative model buys you

p_\text{model}(\boldsymbol{x}) \qquad \text{and} \qquad p_\text{model}(\boldsymbol{h} \mid \boldsymbol{x})

$p_\text{model}(\boldsymbol{x})$	a model of the DATA itself — it can answer inference questions about the relationships among the input variables in x, and reason about them under uncertainty	density
$p_\text{model}(\boldsymbol{h}\mid\boldsymbol{x})$	a REPRESENTATION of x — and by taking expectations of h at different layers of the hierarchy, many different representations at many levels of abstraction	posterior

Why this is the goal, not just a technique

A discriminative model learns one mapping — $\boldsymbol{x}\!\to\!y$ . A generative model with latents learns the joint structure of the world it was shown, so the same model can fill in missing inputs, flag anomalies, generate new examples, and hand higher layers a representation organized by the causes behind the data. That is why so much of this book’s Part III machinery exists: it is the promise of AI systems that hold the many intuitive concepts they need and reason about them in the face of uncertainty.

Part III, assembled

This chapter did not invent its tools in isolation — it spent the four chapters before it building them. The deep generative models here stand on all four at once:

The graph language of chapter 16 says what the models are; the Monte Carlo methods of chapter 17 let us sample them; the partition-function tools of chapter 18 let us train the undirected ones; and the variational inference of chapter 19 handles their intractable posteriors. Chapter 20 is where all four combine.

We hope you will find new ways to make these approaches more powerful — and continue the journey to understanding the principles that underlie learning and intelligence. That closes Part III, and with it this interactive edition. From here you can revisit the full table of contents or browse every term in the glossary.

Check yourself — evaluating generative models & conclusion

1.Model A reports a stochastic ESTIMATE of its log-likelihood; model B reports a deterministic LOWER BOUND. A scores higher. Which is the better model?

2.You use AIS to estimate log Z so you can report log p̃(x) − log Z for your new model. How can this mislead you?

3.Why is even a tiny change in data preprocessing unacceptable when comparing generative models, unlike in object recognition?

4.A generative model produces gorgeous samples. Why is that NOT proof it's a good model?

5.Why does the text say there is no single 'right' evaluation metric, and how does this connect to the two KL directions?

5 questions