The last big idea rescuing generalization in high dimensions: maybe the data doesn’t fill its space at all. A manifold is a connected set of points, each with a neighborhood, that locally looks like Euclidean space — we live on one: the Earth’s surface feels like a 2-D plane but is a spherical manifold in 3-D. Neighborhoods imply transformations for moving along the manifold (walk north, south, east, west). Machine learning uses the word loosely: a connected point set well-approximated by few degrees of freedom embedded in a higher-dimensional space, each degree a local direction of variation — with dimensionality allowed to vary (a figure-eight is 1-D except at its 2-D crossing).
Fig 5.11 (recreated) — data concentrated near a 1-D manifold, like a twisted string in the plane. The solid line is what the learner should infer.
Manifold learning algorithms assume most of consists of invalid inputs: interesting inputs live only along a collection of manifolds, with interesting output variation only along manifold directions (or when hopping between manifolds). Introduced for continuous unsupervised settings, the probability-concentration idea generalizes to discrete and supervised cases. Is the manifold hypothesis manifold hypothesis Real data (images, text, sound) concentrates near low-dimensional connected manifolds in the embedding space — so learn manifold coordinates, not ℝⁿ coordinates. defined in ch. 5 — open in glossary right? For AI tasks — images, sounds, text — two lines of evidence say approximately yes:
Figs 5.12–5.13 (stylized) — evidence 1: real-data distributions are HIGHLY concentrated (uniform noise essentially never resembles structured input; random letters essentially never form English). Evidence 2: we can exhibit the transformations connecting examples.
What this buys — manifold coordinates
When data hugs a low-dimensional manifold, the natural representation is coordinates on the manifold, not coordinates in — as we give addresses by house number along a 1-D road, not by 3-D position. Extracting those coordinates is hard, but it is the promise behind much of representation learning: fig 5.13’s face dataset varies along exactly two pose angles, and by ch. 20 the book shows algorithms that discover and disentangle such coordinates (fig 20.6).
That concludes Part I — the mathematical and machine-learning foundations. Linear algebra gave the objects, probability the uncertainty, numerical computation the algorithms, and this chapter the learning framework plus three reasons simple methods hit walls on AI tasks. Everything is in place: on to deep learning proper — chapter 6, Deep Feedforward Networks.
Check yourself — manifolds
1.What does 'manifold' mean in machine learning usage?
2.What is the first line of evidence for the manifold hypothesis?
3.And the second line of evidence?
4.What representational lesson does the roads-and-addresses analogy teach?