§14.6–14.7Learning Manifolds with Autoencoders · Contractive Autoencoders

§14.6 Learning manifolds with autoencoders

Like many learning algorithms, autoencoders exploit the manifold hypothesis manifold hypothesis Real data (images, text, sound) concentrates near low-dimensional connected manifolds in the embedding space — so learn manifold coordinates, not ℝⁿ coordinates. defined in ch. 5 — open in glossary (§5.11.3): data concentrates near a low-dimensional manifold (or a small set of them). But where most algorithms merely behave correctly on the manifold, autoencoders aim to learn its structure.

The key characterization of a manifold is its set of tangent planes tangent plane At a point x on a d-dimensional manifold, the plane spanned by the d basis vectors of local variation — the directions in which x can move infinitesimally while staying on the manifold. defined in ch. 14 — open in glossary : at a point $\boldsymbol{x}$ on a $d$-dimensional manifold, $d$ basis vectors span the local directions of variation allowed on the manifold — how you can nudge $\boldsymbol{x}$ infinitesimally while staying on it. The book’s fig 14.6 makes this concrete with a translated MNIST digit: vertical translation traces a 1-D curved manifold through 784-dimensional pixel space, and the tangent vector at each point is an image (which pixels brighten, which darken as the digit shifts). Revisit the geometry interactively — the tangent/normal split below is exactly the concept:

The two forces. Every autoencoder training procedure is a compromise:

The one-dimensional picture (fig 14.7) shows the compromise vividly — an optimal reconstruction that is invariant near data and steep between:

Embeddings, and the non-parametric alternative. A representation of a point on (or near) the manifold is its embedding embedding A representation of a data point on (or near) a manifold: a low-dimensional vector with fewer dimensions than the ambient input space. Non-parametric manifold methods learn one per example; encoders map any point to its embedding. defined in ch. 14 — open in glossary — a vector with fewer dimensions than the “ambient” input space. Early manifold learning was mostly non-parametric: build a nearest-neighbor graph over training examples, associate each node with a tangent plane spanned by the difference vectors to its neighbors, then stitch a global coordinate system by optimization or a linear system — even a density, by tiling the manifold with flat Gaussian “pancakes” (manifold Parzen windows):

The fundamental difficulty (Bengio & Monperrus 2005): these methods generalize only by interpolating between neighbors. If the manifold is not smooth — many peaks, troughs, twists — you need training examples covering every one. And AI manifolds are like that: watch one pixel $x_i$ as an image translates, and its value oscillates once per peak of brightness in the template. The complexity of the image’s brightness pattern drives the complexity of the manifold. This is the case for distributed representations and deep learning for manifold capture — parametric encoders that share structure across the whole space instead of memorizing local patches.

§14.7 Contractive autoencoders

The contractive autoencoder contractive autoencoder (cae) An autoencoder penalizing its encoder's Jacobian, Ω(h) = λ‖∂f(x)/∂x‖²_F, so input neighborhoods contract; the few directions with singular values above 1 become the learned manifold tangents. Weights of f and g must be tied to block the ε-scaling cheat. defined in ch. 14 — open in glossary (Rifai et al. 2011) regularizes the code $\boldsymbol{h} = f(\boldsymbol{x})$ explicitly, demanding the encoder’s derivatives be as small as possible:

The name is earned: trained to resist input perturbations, the CAE maps a neighborhood of inputs to a smaller neighborhood of codes — it contracts locally. In the linear-operator view, the Jacobian $\boldsymbol{J}$ at $\boldsymbol{x}$ approximates $f$ as linear, and a linear operator is formally contractive if it keeps $\lVert \boldsymbol{J}\boldsymbol{x} \rVert \le 1$ for every unit vector — if it shrinks the unit sphere. Watch a 2-D “Jacobian” act on the unit circle — the CAE’s ideal is an ellipse squashed in every direction except the manifold tangent:

The DAE sibling. Alain & Bengio (2013): in the limit of small Gaussian input noise, denoising reconstruction error is equivalent to a contractive penalty on the reconstruction function $g \circ f$. In other words: DAEs make the reconstruction resist small-but-finite perturbations; CAEs make the feature extraction $f$ resist infinitesimal ones. (When pretraining features for a classifier, penalizing $f$ — as the CAE does — usually beats penalizing $g \circ f$; a contractive penalty on $f$ also connects to score matching, §14.5.1.) A useful footnote on locality: the CAE is contractive only at training points — globally, two distant inputs can map to codes farther apart, and $f$ plausibly expands in the gaps between manifolds (recall fig 14.7’s steep segments). With sigmoidal code units, an easy way to shrink the Jacobian is to saturate — so CAEs tend toward near-binary codes spread across the hypercube’s corners.

Two practical issues. Deep CAEs make the Jacobian penalty expensive; Rifai et al. instead train a stack of single-layer CAEs, each reconstructing the previous layer’s code, and compose them — not identical to a joint deep penalty, but locally contractive layer by layer, hence contractive as a whole. And the penalty has a degenerate escape: an encoder that just multiplies by a tiny $\epsilon$ (with a decoder dividing by $\epsilon$) drives $\Omega(\boldsymbol{h}) \to 0$ while reconstructing perfectly — having learned nothing. The fix is to tie the weights: both $f$ and $g$ are affine layers with elementwise nonlinearities, so set $g$‘s weight matrix to the transpose of $f$‘s and the rescaling cheat becomes impossible.

Next: a parametric encoder trained to predict the outcome of sparse-coding inference, and where autoencoders earn their keep in practice — predictive sparse decomposition & applications.