§14.0–14.3Autoencoders · Undercomplete & Regularized · Depth

§14.0 A network that copies its input — on purpose, imperfectly

An autoencoder autoencoder An encoder plus decoder trained to preserve information while giving the representation useful properties. defined in ch. 1 — open in glossary is a neural network trained to copy its input to its output through an internal code $\boldsymbol{h}$: an encoder $\boldsymbol{h} = f(\boldsymbol{x})$ followed by a decoder that produces a reconstruction $\boldsymbol{r} = g(\boldsymbol{h})$:

If it learns $g(f(\boldsymbol{x})) = \boldsymbol{x}$ everywhere, it is useless. Autoencoders are deliberately designed to be unable to copy perfectly — restricted so they copy only approximately, and only inputs resembling the training data. Forced to prioritize which aspects of the input to preserve, the model often learns useful properties of the data as a byproduct. Modern treatments also generalize $f$ and $g$ to stochastic mappings $p_{\text{encoder}}(\boldsymbol{h} \mid \boldsymbol{x})$ and $p_{\text{decoder}}(\boldsymbol{x} \mid \boldsymbol{h})$ — that story starts in §14.4.

The idea is decades old (LeCun 1987; Bourlard & Kamp 1988; Hinton & Zemel 1994), traditionally serving dimensionality reduction and feature learning; recent theoretical links to latent-variable models moved autoencoders to the frontier of generative modeling (chapter 20). An autoencoder is just a special case of a feedforward network — trained the same way, with minibatch SGD on back-propagated gradients — though uniquely it can also be trained by recirculation recirculation An autoencoder-specific learning algorithm comparing the network's activations on the original input with those on the reconstruction; regarded as more biologically plausible than back-propagation but rarely used. defined in ch. 14 — open in glossary , comparing activations on the original input against activations on the reconstruction; more biologically plausible than backprop, and rarely used.

§14.1 Undercomplete autoencoders

We usually don’t care about the decoder’s output — we train the copy task hoping $\boldsymbol{h}$ inherits useful properties. The classic constraint: give $\boldsymbol{h}$ smaller dimension than $\boldsymbol{x}$. Such an autoencoder is undercomplete undercomplete autoencoder An autoencoder whose code dimension is smaller than its input dimension, forcing the copy task to keep only the most salient features of the data; with a linear decoder and MSE it learns PCA's principal subspace. defined in ch. 14 — open in glossary ; unable to store everything, it must capture the most salient features of the training data. Learning is simply minimizing:

Power cuts both ways. Give the encoder and decoder too much capacity and the copy task teaches nothing — in the extreme, a one-dimensional code with a powerful enough nonlinear encoder could memorize an index $i$ for each training example $\boldsymbol{x}^{(i)}$, and the decoder could map indices back:

§14.2 Regularized autoencoders

The same failure strikes codes of dimension equal to the input, and the overcomplete case (code dimension greater than the input) — there even a linear encoder and decoder can copy without learning anything. Ideally we would pick code size and capacity freely, to match the complexity of the distribution being modeled. Regularized autoencoders regularized autoencoder An autoencoder whose loss demands properties beyond copying — sparse codes, small derivatives, robustness to noise — so that even an overcomplete, high-capacity model learns data structure instead of the identity function. defined in ch. 14 — open in glossary make that possible: instead of capping capacity, the loss demands properties beyond copying — sparse representations, small derivatives of the representation, robustness to noise or missing inputs. A nonlinear, overcomplete, regularized autoencoder can still learn something useful even with capacity enough for the trivial identity.

(One more route, developed in chapter 20: nearly any generative model with latent variables and an inference procedure — descendants of the Helmholtz machine like the variational autoencoder and generative stochastic networks — acts as a high-capacity autoencoder whose encodings are useful without regularization, because it was trained to maximize the probability of the data rather than to copy.)

§14.2.1 Sparse autoencoders

A sparse autoencoder sparse autoencoder An autoencoder trained on L(x, g(f(x))) + Ω(h) with Ω(h) = λΣ|hᵢ|; equivalent to approximate maximum-likelihood training of a generative latent-variable model with a Laplace prior on h. Rectified linear codes give actual zeros. defined in ch. 14 — open in glossary adds a sparsity penalty $\Omega(\boldsymbol{h})$ on the code to the reconstruction error:

…but there is a principled reading: the sparse autoencoder approximates maximum likelihood in a generative model with latent variables. Watch the penalty emerge from a Laplace prior over the code:

This view supplies a deeper motivation: training a sparse autoencoder is a way of approximately training a generative model, and the learned features are useful because they describe the latent variables that explain the input. Two footnotes from the book: early work (Ranzato et al. 2007, 2008) connected the sparsity penalty to the $\log Z$ term of undirected models $p(\boldsymbol{x}) = \tfrac{1}{Z}\tilde p(\boldsymbol{x})$ — minimizing $\log Z$ stops a model from having high probability everywhere, as sparsity stops an autoencoder from having low reconstruction error everywhere (an intuitive, not mathematical, correspondence). And to get actual zeros in $\boldsymbol{h}$, use rectified linear code units (§6.3.1) with a zero-pushing prior — the average number of zeros becomes directly controllable (Glorot et al. 2011).

§14.2.2 Denoising autoencoders — change the task, not the cost

Instead of adding a penalty, change the reconstruction term itself. Where a traditional autoencoder minimizes $L(\boldsymbol{x}, g(f(\boldsymbol{x})))$ (eq 14.8) — happy to become an identity — a denoising autoencoder denoising autoencoder (dae) An autoencoder fed a corrupted x̃ ~ C(x̃|x) and trained to reconstruct the CLEAN x — it must undo corruption rather than copy. With Gaussian corruption and squared error, its reconstruction-minus-input field g(f(x)) − x estimates the score ∇ₓ log p(x). defined in ch. 14 — open in glossary minimizes:

Denoising lets overcomplete, high-capacity models serve as autoencoders safely — the identity function is no longer a solution. The full treatment (corruption processes, the learned score field) is §14.5, next.

§14.2.3 Contractive autoencoders — penalize the derivatives

A third strategy keeps the penalty form $L + \Omega$ but aims $\Omega$ at the code’s derivatives:

This is the contractive autoencoder contractive autoencoder (cae) An autoencoder penalizing its encoder's Jacobian, Ω(h) = λ‖∂f(x)/∂x‖²_F, so input neighborhoods contract; the few directions with singular values above 1 become the learned manifold tangents. Weights of f and g must be tied to block the ε-scaling cheat. defined in ch. 14 — open in glossary (CAE), with deep connections to denoising autoencoders, manifold learning and probabilistic modeling — unpacked in §14.7.

The four identity-blockers, side by side

§14.3 Representational power, layer size and depth

Autoencoders are often trained with single-layer encoders and decoders — but nothing requires it, and depth pays. Encoder and decoder are each themselves feedforward networks, so each individually inherits every advantage of depth from §6.4.1. The universal approximator theorem says one hidden layer can represent the identity along the data’s domain arbitrarily well — but with a shallow input-to-code map, we cannot enforce arbitrary constraints (say, sparsity) on the code; a deep autoencoder, with at least one extra hidden layer inside the encoder, can approximate any input-to-code mapping given enough units. Depth can also exponentially reduce both the computational cost and the training data needed to represent some functions — and experimentally, deep autoencoders compress much better than shallow or linear ones (Hinton & Salakhutdinov 2006).

The common training strategy is itself a reason shallow autoencoders matter: greedily pretrain a stack of shallow autoencoders (each learning to encode the previous one’s code), then compose them into the deep autoencoder — the recipe that returns in chapter 15’s treatment of layer-wise pretraining.

Next: encoders and decoders become distributions, and denoising training turns out to estimate the gradient of the data’s log-density — stochastic autoencoders & the denoising score field.