Other Forms of Learned Inference · Deep Learning Interactive

§19.5.2 Learning to infer, several ways

Wake-sleep was one way to learn inference; the idea has spread. The common thread: replace the slow, iterative optimization for $q$ with a network that outputs it in a single forward pass — trading a little accuracy for a large speedup.

The trade at the heart of learned inference. Left: classical variational inference iterates mean-field fixed-point updates to convergence for every input. Right: a learned inference network maps v → q in ONE pass — amortizing the cost of inference across all future inputs.

Four forms of learned approximate inference. Click a cell for detail.
	Approach	How it learns to infer
DBM amortized inference	DBM inference network	run the net, apply ONE mean-field step to refine, then train the net to output that refined estimate
Predictive sparse decomposition	PSD (§14.8)	train a shallow encoder to predict the sparse code — learned approximate MAP inference
ISTA / deep encoders	ISTA deep encoder	a deep encoder implements the competitive, iterative inference a shallow one cannot
Variational autoencoder	variational autoencoder	the inference net defines L; train its parameters to increase L — no targets needed

Dotted-underlined cells have explanations — click one.

The last row is the one that took over. The variational autoencoder needs no constructed targets at all: its inference network doesn’t try to match some externally-computed $q^*$ — it is part of the ELBO’s definition, and gradient ascent on $\mathcal{L}$ trains encoder and model together.

Where this chapter has brought us

We started by recasting inference as maximizing the ELBO $\mathcal{L}$ , then made it cheap in three escalating ways: restrict $q$ (mean field), point-estimate $q$ (MAP), and finally learn the whole inference map with a network. Together with chapter 18’s partition-function tools, this is the machinery that makes the otherwise-intractable deep generative models of the next chapter — deep belief networks, deep Boltzmann machines, and variational autoencoders — trainable at all.

Check yourself — other forms of learned inference

1.What is the common idea behind all forms of learned approximate inference?

2.How does a learned inference network for a DBM (Salakhutdinov & Larochelle 2010) get trained?

3.Why can't PSD's shallow encoder do what a deep (ISTA-style) encoder can?

4.What makes the variational autoencoder's approach to learned inference distinctive?

5.This chapter made inference cheap in three escalating ways. What are they, in order of increasing approximation?

5 questions