§15.6 The catalog of clues
To close the chapter, back to the original question one last time: what makes one representation better than another? The answer of §15.3: an ideal representation disentangles the underlying causal factors — especially those relevant to our applications. Most representation-learning strategies work by introducing clues that help the learner find and separate those factors. Supervised learning is the strongest clue — a label delivered with each usually specifies at least one factor of variation directly. To exploit abundant unlabeled data, learners rely on less direct hints: implicit prior beliefs that we, the algorithm’s designers, impose. The no-free-lunch theorem says regularization of some kind is necessary, and no universally superior choice exists — deep learning’s wager is that a set of fairly generic priors covers the kinds of tasks people and animals solve.
Each entry, with where you have already met it in this book:
| What it assumes | Where it appeared | |
|---|---|---|
| Smoothness | f(x + εd) ≈ f(x) — generalize to nearby points | nearly every algorithm; §5.11.2 |
| Linearity | some variable relationships are linear — predict far from the data | linear models; §7.13 |
| Multiple explanatory factors | data generated by several underlying factors; tasks easy GIVEN the factors | the §15.3 semi-supervised argument; §15.4 directions |
| Causal factors | the learned h are CAUSES of x, not effects | §15.3 (Schölkopf robustness) |
| Depth / hierarchy | abstract concepts defined via simpler ones; a multi-step program | §6.4.1, §15.5 |
| Shared factors across tasks | each task's yᵢ ties to a SUBSET of a common factor pool h | multi-task §7.7; transfer §15.2 |
| Manifolds | probability mass concentrates on low-dimensional, locally connected regions | §5.11.3, §13.5, §14.6 (autoencoders learn them explicitly) |
| Natural clustering | each CONNECTED manifold carries a single class | tangent prop & manifold tangent classifier §7.14; adversarial training §7.13 |
| Temporal & spatial coherence | important factors change slowly over time (or are at least easy to predict) | slow feature analysis §13.3 |
| Sparsity | most features are irrelevant to most inputs — "present/absent" features should mostly be absent | §7.10, §13.4, §14.2.1 |
| Simplicity of factor dependencies | high-level factors relate through SIMPLE (marginal-independent, linear, or shallow) dependencies | factorial priors §13.1/§14.2; linear classifiers on h §15.4 |
One catalog, one book
Read the rows again and this catalog is a map of everything so far: smoothness smoothness prior f(x) ≈ f(x+ε): the implicit prior of k-NN/local kernels/trees — needs O(k) examples for O(k) regions and can't extrapolate a checkerboard; DL's composition assumption gets O(2ᵏ). defined in ch. 5 — open in glossary from Part I, sparsity and the manifold hypothesis manifold hypothesis Real data (images, text, sound) concentrates near low-dimensional connected manifolds in the embedding space — so learn manifold coordinates, not ℝⁿ coordinates. defined in ch. 5 — open in glossary from the regularization and autoencoder chapters, the slowness principle slowness principle The idea that important characteristics of a scene change SLOWLY compared with raw measurements (a zebras stripes flicker a pixel fast, but zebra present does not) — so regularize features to change slowly over time (penalty λΣₜ L(f(x⁽ᵗ⁺¹⁾),f(x⁽ᵗ⁾))). defined in ch. 13 — open in glossary from linear factor models, depth from the feedforward chapters, shared factors from multi-task and transfer learning, causal disentangling from this chapter. Representation learning ties together all the many forms of deep learning — feedforward and recurrent networks, autoencoders and deep probabilistic models all learn and exploit representations. Learning the best possible one remains an open and exciting research problem.
That closes Chapter 15 — and sets up the rest of Part III: chapters 16–19 build the machinery of structured probabilistic models, sampling, partition functions and inference, so that chapter 20’s deep generative models can actually learn the causal factors this chapter argued for.
Check yourself — clues to underlying causes
1.Why must representation learning rely on "clues" (implicit priors) at all?
2.Smoothness and linearity are often conflated. How do they actually differ?
3.Which catalog priors do slow feature analysis and adversarial training respectively instantiate?
4.What does the "natural clustering" prior assert?
5.The "simplicity of factor dependencies" prior underlies which everyday modeling choice?