Predictive Sparse Decomposition · Applications of Autoencoders

§14.8 Predictive sparse decomposition

Predictive sparse decomposition (PSD; Kavukcuoglu et al. 2008) is the deliberate hybrid of the chapter’s two encoder philosophies: sparse coding’s optimization-as-encoder (§13.4) and the autoencoder’s parametric map. A parametric encoder $f(\boldsymbol{x})$ is trained to predict the output of iterative inference. Both $f$ and the decoder $g(\boldsymbol{h})$ are parametric; during training, $\boldsymbol{h}$ itself is controlled by the optimization algorithm. The objective carries one term per force:

The PSD objective (eq 14.19)

\lVert \boldsymbol{x} - g(\boldsymbol{h}) \rVert^2 \;+\; \lambda \lvert \boldsymbol{h} \rvert_1 \;+\; \gamma \lVert \boldsymbol{h} - f(\boldsymbol{x}) \rVert^2

$\lVert \boldsymbol{x} - g(\boldsymbol{h}) \rVert^2$	reconstruction: the code must explain the input through the decoder — sparse coding's first force	reconstruct
$\lambda \lvert \boldsymbol{h} \rvert_1$	sparsity: the familiar L¹ pull toward exact zeros — sparse coding's second force	sparsify
$\gamma \lVert \boldsymbol{h} - f(\boldsymbol{x}) \rVert^2$	the NEW term: the code must stay near what the parametric encoder predicts — simultaneously teaching f to imitate inference and keeping inference easy	predictable

Why the third term changes everything

Training alternates minimization over $\boldsymbol{h}$ and over the parameters, as in sparse coding — but the $\boldsymbol{h}$ -step is now fast: $f(\boldsymbol{x})$ hands it a good initial value, and the $\gamma$ tether keeps the answer nearby, so simple gradient descent gets reasonable codes in as few as ten steps. Subtler and more important: this is not the same as training sparse coding first and then fitting $f$ to its codes. Joint training regularizes the decoder to use parameters for which $f$ can infer good codes — the dictionary itself is shaped to be predictable. PSD is an early instance of learned approximate inference, and the tools of §19.5 reinterpret it as maximizing a lower bound on the log-likelihood of a directed sparse-coding model.

At deployment the iterative optimization disappears entirely: the parametric encoder $f$ computes the features — cheap next to gradient-descent inference, and differentiable, so PSD models can be stacked and used to initialize a deep network trained with another criterion. It has served unsupervised feature learning for object recognition in images and video, and for audio. Where the pieces land:

Three encoder philosophies, one family. Click a cell for the fine print.
	Sparse coding (§13.4)	PSD (§14.8)	Parametric AE (§14.1–14.7)
Encoder	optimization (argmin per input)	both: optimization in training, f(x) at test	parametric map f(x)
Test-time cost	slow — iterative inference per input	one forward pass	one forward pass
Encoder generalization error	none (convex inference)	some — f approximates inference	yes — learned map
Backprop / stackable	hard	yes — f is differentiable	yes

Dotted-underlined cells have explanations — click one.

§14.9 Applications of autoencoders

Dimensionality reduction was one of the first applications of representation learning, and an early motivation for autoencoders themselves. The emblematic result (Hinton & Salakhutdinov 2006): pretrain a stack of RBMs, use their weights to initialize a deep autoencoder with gradually smaller layers ending in a 30-unit bottleneck — the resulting code reconstructs better than 30-D PCA, and its representation is qualitatively easier to interpret, with categories appearing as well-separated clusters. Lower-dimensional representations help downstream tasks generally: less memory and runtime, and — since many reductions place semantically related examples near each other — the mapping itself aids generalization.

The task that benefits most is information retrieval: finding database entries resembling a query. Beyond the usual gains, low dimensions unlock something qualitatively new — make the code low-dimensional and binary, and search becomes a hash lookup. This is semantic hashing (Salakhutdinov & Hinton 2007, 2009), applied to text and to images:

Semantic hashing — a deep encoder with saturated sigmoid outputs maps each entry to a near-binary code; thresholding gives a bucket address in a hash table. A query retrieves its whole bucket in O(1), and flipping single bits of the query’s code walks to the slightly-less-similar neighboring buckets.

Getting binary codes out of a differentiable network takes a trick. Use sigmoids on the final encoding layer and train them to saturate near 0 or 1 for all inputs: inject additive noise just before the sigmoid, with magnitude increasing over training. To push information through despite the noise, the network must amplify the sigmoid’s inputs — until saturation. The idea of learning the hash function has been pushed further, including losses tied directly to finding nearby examples in the table (Norouzi & Fleet 2011).

Chapter close. Autoencoders began as the copy machine that must not copy — and ended as score estimators, manifold learners, inference approximators and retrieval engines. Chapter 15 asks the bigger question these tools serve: what makes one representation better than another?

Check yourself — PSD & autoencoder applications

1.Predict the outcome — delete the γ‖h − f(x)‖² term from PSD's objective. What model are you left with?

2.Why is PSD's per-example minimization over h fast (≈ ten gradient steps)?

3.How does PSD differ from simply training a sparse coding model, then fitting f(x) to predict its codes?

4.At deployment, what does a PSD model actually run?

5.Semantic hashing needs the code sigmoids saturated at ≈0/≈1. What is the training trick, and why does it work?

5 questions