§7.14 Regularizing along the manifold
Several of this chapter’s ideas share a hidden theme: the manifold hypothesis (§5.11.3) — data concentrates near a low-dimensional manifold, and examples on the same manifold share a class. If we know the directions of variation along that manifold (translating, rotating, or scaling an image keeps its class), we can bake that invariance directly into a model.
The earliest attempt is tangent distance tangent distance A non-parametric nearest-neighbor algorithm whose metric is the distance between the class MANIFOLDS (approximated by their tangent planes at each point) rather than the generic Euclidean distance — invariant to local factors of variation. defined in ch. 7 — open in glossary : a nearest-neighbor classifier that measures distance not with the generic Euclidean metric but with the distance between the manifolds the two points lie on. Since finding the closest pair of points on two manifolds is expensive, it approximates each manifold by its tangent plane at the point and measures the distance between those planes — a small linear system. It needs the tangent vectors specified in advance.
Tangent prop tangent prop Train a net with a penalty Ω(f)=Σᵢ(∇ₓf·v⁽ⁱ⁾)² that keeps each output locally invariant along known manifold tangents v⁽ⁱ⁾ (translation, rotation, scaling). The infinitesimal cousin of dataset augmentation. defined in ch. 7 — open in glossary (fig 7.9) carries the same idea into a neural net: add a penalty that makes each output locally invariant along the known manifold tangents. The output should not change as we move along the manifold (the class is preserved) but is free to change across it. Drag the point and toggle the two styles:
The classifier should be constant along the (the factor of variation that keeps the class) and free to change along the normal (where the class changes).
Formally, penalize the directional derivative of along each tangent:
Term by term
| the known manifold tangent vectors — directions like translation, rotation, scaling that keep the class fixed | invariant dirs | |
| the directional derivative of the output along tangent i; driving it to zero makes ∇f ORTHOGONAL to the tangents — f is flat along the manifold | kill this | |
| sum over tangent directions (and outputs), scaled by a hyperparameter | the penalty |
A family of invariance regularizers
Tangent prop sits in a family. Dataset augmentation (§7.4) encodes the same prior knowledge but by taking finite steps (visiting transformed inputs) rather than an infinitesimal penalty — which is why augmentation confers resistance to larger perturbations and plays nicely with ReLU (different units activate for different transformed versions), while tangent prop’s infinitesimal approach struggles with ReLU. Two more methods drop the “known directions” requirement and instead resist change in all directions:
| mechanism | directions | scale | |
|---|---|---|---|
| Tangent distance | NN metric = distance between class manifolds (via tangent planes) | specified tangents | — |
| Tangent prop | penalize ∇f·v along known tangents | specified | infinitesimal |
| Dataset augmentation | train on transformed inputs | specified | finite |
| Double backprop | penalize the whole Jacobian ‖∇f‖ | all directions | infinitesimal |
| Adversarial training | train f(x′)=f(x) for x′ near x | all directions | finite |
The symmetry is neat: dataset augmentation is the finite version of tangent prop, and adversarial training is the finite version of double backprop double backprop Regularizing the Jacobian ∇ₓf to be small in ALL input directions; adversarial training is its non-infinitesimal counterpart (as dataset augmentation is to tangent prop). defined in ch. 7 — open in glossary .
Learning the tangents instead of specifying them
Tangent prop’s weakness is needing the tangent vectors up front. The manifold tangent classifier manifold tangent classifier Tangent prop WITHOUT hand-specified tangents: an autoencoder first learns the manifold tangent vectors (ch14), which then regularize the classifier — capturing object-specific factors beyond translation/rotation/scaling. defined in ch. 7 — open in glossary (Rifai et al.) removes that: an autoencoder first learns the manifold structure by unsupervised learning and estimates the tangent vectors (chapter 14), which then regularize the classifier exactly as tangent prop does. The learned tangents go beyond the classical geometric invariants (translation, rotation, scaling) to object-specific factors — like the ways a body’s parts can move. The algorithm is just two steps: (1) an autoencoder learns the manifold tangents unsupervised; (2) those tangents regularize the classifier via the tangent-prop penalty.
End of the chapter
That covers the main strategies for regularizing neural networks — norm penalties, constraints, augmentation, noise, early stopping, parameter sharing, sparsity, bagging, dropout, adversarial and tangent methods. Regularization is a central theme of machine learning and will resurface in nearly every chapter that follows. The other central theme — how to actually optimize these models — is next.
On to training: chapter 8, Optimization for Training Deep Models.
Check yourself — tangent methods and the manifold hypothesis
1.What does tangent prop's penalty Ω(f) = Σᵢ(∇ₓf·v⁽ⁱ⁾)² actually enforce?
2.In TangentLab, what do the tangent and normal directions represent, and why should the classifier treat them differently?
3.How are dataset augmentation and tangent prop related, and why does augmentation work better with ReLU?
4.The comparison table pairs four methods by 'directions' and 'scale'. Which pairing is correct?
5.What problem does the manifold tangent classifier solve, and how?