Representation Learning · Greedy Layer-Wise Unsupervised Pretraining

§15.0 What is a representation worth?

This chapter names the question Part III has been circling: what does it mean to learn a representation, and what makes one better than another? It covers how algorithms share statistical strength across tasks — including using unsupervised information for supervised tasks — how shared representations handle multiple domains and transfer to tasks with few or no examples, and finally why any of this works: distributed representations, depth, and assumptions about the data’s underlying causes.

The premise: many tasks are easy or hard depending on how the information is represented — a principle as true of daily life and computer science as of machine learning:

Same information, different representation, different task difficulty.
	Helpful representation	Unhelpful representation
Divide 210 by 6	Arabic numerals — long division exploits the place-value system	Roman numerals: CCX ÷ VI
Insert a number into a sorted collection	red-black tree — O(log n)	linked list — O(n)
Classify an image	top-hidden-layer features — classes (ideally) linearly separable	raw pixels — classes hopelessly entangled

Dotted-underlined cells have explanations — click one.

Generally, a good representation is one that makes a subsequent learning task easier — so the right representation depends on which task comes next.

Every supervised net is secretly two models

A feedforward classifier’s last layer is typically a simple linear (softmax) classifier; everything below it exists to hand that classifier a representation it can work with. Supervised training imposes no explicit conditions on the intermediate features, yet they take on — increasingly toward the top — whatever properties make classification easier, e.g. linear separability that raw inputs lacked. The split is real: the last layer could instead be a nearest-neighbor classifier, and the penultimate features would then learn different properties to suit it.

Other representation learners shape the representation explicitly — e.g. encouraging independent code elements because more-independent distributions are easier to model. Either way there is a main objective and a representation learned as a side effect, and either way one faces the central tradeoff: preserve information about the input vs. attain nice properties (like independence or sparsity).

Why care so much? Semi-supervised learning. Unlabeled data is abundant; labeled data is scarce, and supervised training on a small labeled set overfits badly. Learning good representations from the unlabeled data and then solving the supervised task on top is one route out — and one popular hypothesis for how humans manage to learn from a handful of labeled examples (alongside big ensembles and Bayesian tricks that brains might also use).

§15.1 Greedy layer-wise unsupervised pretraining

Unsupervised learning played the key historical role in reviving deep neural networks: it let researchers train a deep fully-connected supervised network for the first time without architectural crutches like convolution or recurrence. The procedure — greedy layer-wise unsupervised pretraining — is the canonical case of a representation learned for one task (capturing the input distribution’s shape) serving another (supervised learning on the same inputs). It relies on any single-layer representation learner: an RBM, a one-layer autoencoder (ch14), a sparse coding model (§13.4)… Each layer is pretrained unsupervised, consuming the previous layer’s output and producing a new — hopefully simpler — representation:

Algorithm 15.1 — greedy layer-wise unsupervised pretraining

1Given: unsupervised feature learner $L$ ; raw data $X$ (one row per example); $m$ stages; optional fine-tuner $T$ (with targets $Y$ )ⓘ
2 $f \leftarrow$ Identity function
3 $\tilde{X} = X$ ⓘ
4for $k = 1, \ldots, m$ do
5 $f^{(k)} = L(\tilde{X})$ ⓘ
6 $f \leftarrow f^{(k)} \circ f$ ⓘ
7 $\tilde{X} \leftarrow f^{(k)}(\tilde{X})$ ⓘ
8end for
9if fine-tuning then $f \leftarrow T(f, X, Y)$ ⓘ
10Return $f$

Each word of the name is doing work: greedy, because each piece of the solution is optimized independently, one piece at a time, never jointly; layer-wise, because the pieces are the network’s layers — layer $k$ trains while all previous layers stay fixed; unsupervised, because each layer trains with an unsupervised representation learner; and pretraining, because it is only meant to be the first step before a joint fine-tuning phase. (In common usage “pretraining” names the whole two-phase protocol — pretrain, then supervise.) Watch the protocol run:

The two-phase protocol, animated

1Train layer 1 — unsupervised

Stage encoder f⁽¹⁾ = L(X) learns from the raw data alone (an RBM, autoencoder, or sparse coder capturing the input distribution's shape).

step 1 / 4

The 2006 moment

Greedy layer-wise training to sidestep joint deep optimization dates back at least to the Neocognitron (Fukushima 1975). The deep learning renaissance of 2006 began with the discovery that the greedy procedure could find a good initialization for joint training of all layers — and that this trained even fully-connected architectures (Hinton et al. 2006; Bengio et al. 2007; Ranzato et al. 2007). Before that, only convolutional or recurrent depth was considered feasible. Today we know pretraining is not required for deep fully-connected nets — but it was the first thing that worked.

In a supervised context, pretraining acts as a regularizer — in some experiments it decreases test error without decreasing training error — and as a form of parameter initialization. The same trick also initializes other unsupervised models: deep autoencoders (§14.3), and deep probabilistic models with many latent layers — deep belief networks and deep Boltzmann machines (chapter 20). And the greedy idea has a supervised twin: greedy layer-wise supervised pretraining (§8.7.4), built on the premise that a shallow network is easier to train than a deep one.

Next: the harder question — pretraining sometimes helps a lot, often does nothing, and sometimes hurts. When, and why?

Check yourself — representations & greedy pretraining

1.What makes one representation better than another?

2.In what sense does an ordinary supervised feedforward network perform representation learning?

3.Predict the outcome — after Algorithm 15.1's loop finishes (no fine-tuning), what is f, and what happened to layer 1's weights while layer 3 trained?

4.What evidence supports viewing unsupervised pretraining as a REGULARIZER (not just an optimization aid)?

5.Why is representation learning especially attractive for SEMI-supervised problems?

5 questions