§8.7.4–8.7.6Supervised Pretraining · Designing Models to Aid Optimization · Continuation & Curriculum Learning

§8.7.4 Supervised pretraining

Training a complex model on a hard task directly can be too ambitious. A greedy alternative: break the problem into pieces, solve each in isolation (cheaply, if not optimally), then optionally fine-tune the whole thing jointly — a good greedy solution is often a great starting point. The classic form is greedy supervised layer-wise pretraining greedy layer-wise pretraining Pretrain each hidden layer as part of a shallow supervised MLP on the previous layer's output, then stack the layers and jointly fine-tune (Bengio et al. 2007, fig 8.7) — a greedy strategy that gives good intermediate-layer guidance. defined in ch. 8 — open in glossary (Bengio et al., 2007, fig 8.7): train a shallow supervised net, keep its hidden layer, then train a new shallow net on top of that layer’s output — repeating to build depth:

It helps by giving better guidance to the intermediate layers of a deep hierarchy. Related ideas: transfer learning transfer learning Initialize a network with the first k layers of a net trained on a related task/dataset, then jointly fine-tune on the target task (often with fewer examples); §15.2. defined in ch. 8 — open in glossary (initialize with the first $k$ layers of a net trained on a related task, then fine-tune — §15.2), and FitNets, where an easy-to-train wide teacher supplies hints (its middle-layer activations) that a deep thin student is trained to predict, dramatically easing the student’s optimization.

§8.7.5 Designing models to aid optimization

Here is a lesson worth internalizing: the best way to improve optimization is often not to improve the optimizer, but to design a model that is easier to optimize. Most of the past 30 years’ gains came from changing the model family — SGD with momentum from the 1980s still trains state-of-the-art nets. Modern architectures deliberately use near-linear pieces (ReLU, maxout, LSTM) so the gradient flows cleanly and points reliably toward better solutions. Two more tricks shorten the path from the loss to the lower layers:

Skip connections skip connection A connection from layer i to layer i+2 or beyond, easing gradient flow from output toward input. defined in ch. 6 — open in glossary shorten the path from a lower layer’s parameters to the output (mitigating vanishing gradients), and auxiliary heads (as in GoogLeNet) attach extra output copies to intermediate layers so those layers receive a strong gradient directly — an alternative to pretraining, discardable once training finishes.

§8.7.6 Continuation methods and curriculum learning

Many optimization difficulties come from the cost’s global structure. A continuation method continuation method Solve a sequence of increasingly hard objectives J⁽⁰⁾…J⁽ⁿ⁾ (e.g. progressively less-blurred J⁽ⁱ⁾ = E_{θ'~N}J(θ')), each solution initializing the next, to reach a good region of the true cost J. defined in ch. 8 — open in glossary builds a sequence of objectives $J^{(0)}, \dots, J^{(n)}=J$, easy to hard, where each solution initializes the next. The traditional way to make an easier $J^{(i)}$ is to blur the cost (convolve it with a Gaussian): non-convex ripples flatten out, revealing the global basin. Crank up the blur and watch a trapped ball escape its local minimum:

Next: convolutional networks — feedforward nets built for grid-structured data — chapter 9, Convolutional Networks.