When and Why Does Unsupervised Pretraining Work?

§15.1.1 Sometimes a triumph, sometimes a tax

On many classification tasks, greedy layer-wise unsupervised pretraining yields substantial test-error improvements — the observation that re-ignited deep learning in 2006. On many other tasks it confers no benefit or causes noticeable harm: Ma et al. (2015) found that for chemical activity prediction, pretraining was slightly harmful on average yet significantly helpful for many individual tasks. So the practical question is when and why it works.

(Scope note: this analysis targets greedy unsupervised pretraining specifically. Other semi-supervised paradigms exist — virtual adversarial training (§7.13), and single-stage approaches that train the supervised and unsupervised objectives simultaneously as one explicit sum, like the discriminative RBM and the ladder network.)

Pretraining combines two different ideas:

Idea 1 — initialization as a regularizer (the murky one)

The choice of initial parameters can strongly regularize a deep model. The original story — “pretraining picks a better local minimum’s basin” — has not survived: local minima are no longer considered the serious obstacle, and standard training usually does not arrive at a critical point of any kind. The refined story: pretraining may place parameters in a region otherwise inaccessible — say, surrounded by zones where the cost varies so much between examples that minibatch gradients turn to noise, or where Hessian conditioning is so poor that steps must crawl. But our ability to characterize what is retained of the pretrained parameters through supervised training is limited — one reason modern approaches either train both objectives simultaneously, or freeze the features and only fit a classifier on top.

Idea 2 — features for one task serve another (the clearer one)

Learning about the input distribution can help learn the input→output mapping. Train a generative model of car and motorcycle images and it must learn about wheels — and if we are fortunate, its wheel representation is in a form the supervised learner can exploit. This is not yet understood mathematically, so we cannot reliably predict which tasks benefit; much depends on the models — a linear classifier on top of frozen features needs the classes linearly separable in those features, which happens often but not always. (Simultaneous training again looks attractive: the output layer’s constraints shape the features from the start.)

When does it help?

The evidence, condensed. Click a cell for the source or caveat.
	Verdict	Why
Initial representation is poor (words!)	helps — the flagship case	one-hot vectors carry zero similarity information; learned embeddings encode it as distance
Few labels, many unlabeled examples	helps — regularizer view	the added information comes from unlabeled data
Very complicated target function	helps — the right prior	unlike weight decay, pretraining does not bias toward SIMPLE functions — it biases toward features shaped by the input distribution's regularities
Deeper networks	helped more (2010 evidence)	mean AND variance of test error most reduced for deeper nets (Erhan et al. 2010)
Medium labeled sets (CIFAR-10, MNIST)	no longer helps	supervised training with dropout/batch-norm outperforms it at ~5,000 labels per class
Extremely small labeled sets	loses to Bayesian methods	e.g. the alternative splicing dataset (Srivastava 2013)

Dotted-underlined cells have explanations — click one.

Worked example — why one-hot geometry is useless

Take any two distinct one-hot word vectors, $\boldsymbol{e}_i$ and $\boldsymbol{e}_j$ ( $i \neq j$ ). Their squared $L^2$ distance is $\lVert \boldsymbol{e}_i - \boldsymbol{e}_j \rVert^2 = 1^2 + (-1)^2 = 2$ — identically, for every pair. “cat” is exactly as far from “kitten” as from “carburetor”. A learned embedding breaks this tie, and that broken tie is the information pretraining adds.

What actually happens — the function-space picture

Erhan et al. (2010) ran the decisive experiments. Pretraining can even improve training error (deep autoencoders reconstruct better train and test — an optimization effect), but the deepest finding is geometric. Training is non-deterministic: each run halts at a different function — where gradients vanish, where early stopping intervenes, or where steps become impractical. Projecting many runs into function space:

Fig 15.1 (recreated, after Erhan et al. 2010) — learning trajectories of many networks in FUNCTION space (Isomap projection of concatenated outputs; color deepens with training time). All runs start near the center — functions giving roughly uniform class distributions. Pretrained runs (green) consistently terminate in one small region; non-pretrained runs (blue) in another, larger, NON-overlapping region. Since Isomap roughly preserves volumes, the smaller pretrained region suggests reduced estimator variance — the regularization made visible.

Pretrained networks consistently halt in the same small region of function space; non-pretrained networks in a different, larger, non-overlapping one. Pretraining initializes parameters into a region they do not escape — outcomes become more consistent and less likely to be very bad: variance reduction, which is regularization. How exactly? One hypothesis — pretraining encourages features related to the underlying causes of the data — is the bridge to §15.3.

The costs, and where it stands today

Two-phase training exacts real prices. Most regularizers expose a single strength knob; pretraining offers none — either you initialize from the pretrained parameters or you don’t. Simultaneous training has exactly such a knob (the coefficient on the unsupervised cost). And each phase has its own hyperparameters, with phase 1’s quality only measurable after phase 2 runs — the principled fix (select phase-1 hyperparameters by phase-2 validation error; Larochelle et al. 2009) is expensive, so in practice things like the number of pretraining iterations are set cheaply by early stopping on the unsupervised objective.

Where pretraining lives on

Today unsupervised pretraining is largely abandoned — except in natural language processing, where one-hot word representations convey no similarity and unlabeled corpora run to billions of words: pretrain once at scale, then reuse or fine-tune the representation for label-poor supervised tasks (Collobert & Weston 2008; Turian et al. 2010; Collobert et al. 2011 — and recall word embeddings, §12.4). Its idea spread everywhere: supervised pretraining (§8.7.4) with ImageNet-trained conv nets became the standard transfer recipe, with practitioners publishing trained parameters exactly as pretrained word vectors are published. A milestone, absorbed.

Next: transfer made explicit — shared factors between tasks, adapting across domains, and learning with one or zero labeled examples — transfer learning & domain adaptation.

Check yourself — when & why pretraining works

1.Unsupervised pretraining combines which two distinct ideas?

2.What does fig 15.1's function-space visualization show, and what does it suggest?

3.Why are WORD one-hot vectors the flagship case for pretraining — and images a weak one?

4.Compared with training supervised + unsupervised objectives SIMULTANEOUSLY, what control does two-phase pretraining lack?

5.What is unsupervised pretraining's status today, per the book?

6.Why is the old "pretraining picks a better local minimum" explanation no longer accepted?

6 questions