§12.1–12.1.4Large-Scale Deep Learning · CPU · GPU · Distributed · Compression

§12.1 Large-scale deep learning

Deep learning rests on connectionism: an individual unit is not intelligent, but a large population acting together can be. That is why size is paramount — one of the biggest drivers of neural networks’ rising accuracy from the 1980s to today was the exponential growth in network size (yet artificial nets are still only as large as an insect’s nervous system). Big networks demand serious hardware and software.

§12.1.1 Fast CPU implementations

Careful CPU specialization pays off. In 2011, a tuned fixed-point implementation beat a strong floating-point system threefold (Vanhoucke 2011) — though which is faster depends on the CPU. Other wins: cache-friendly data layout and vector instructions. Researchers who neglect these details let implementation speed cap model size, which caps accuracy.

§12.1.2 GPU implementations

Graphics hardware was built to transform many vertices and shade many pixels in parallel, with simple, branch-free arithmetic over massive memory buffers. Neural networks need exactly the same profile — large buffers of parameters, activations, and gradients (too big for cache, so memory bandwidth is the bottleneck), updated in parallel with little control flow:

General-purpose GPUs (GP-GPUs) with CUDA made this programmable, and deep learning adopted them fast. Because efficient GPU code is hard, the practical workflow is to build models as calls to a library of tuned operations (convolution, matrix multiply) — Theano, TensorFlow, Torch — so the same program runs on CPU or GPU without rewriting kernels.

§12.1.3 Large-scale distributed training

When one machine isn’t enough, distribute the work. Data parallelism sends different examples to different machines — trivial for inference. Model parallelism splits a single datapoint’s model across machines. Training-data parallelism is harder: standard SGD is sequential (step $t$ needs the parameters from step $t-1$). The fix is asynchronous SGD asynchronous sgd Lock-free parallel SGD: several cores read shared parameters, compute a gradient, and update without locks. Some updates overwrite each other, but the higher step rate speeds learning overall. Scaled across machines via a parameter server (Dean 2012). defined in ch. 12 — open in glossary — workers read the shared parameters from a parameter server parameter server A machine (or set of machines) that holds the shared model parameters in distributed asynchronous SGD; workers read parameters, compute gradients, and push updates back. defined in ch. 12 — open in glossary , compute a gradient, and push an update without a lock. Drag the worker count to see the tradeoff:

§12.1.4 Model compression

For deployment, inference cost usually matters far more than training cost — train once on a cluster, serve billions of users on resource-constrained devices (a phone). Model compression model compression Replacing an expensive model (often a large regularized net or an ensemble) with a smaller model trained to mimic its function f(x) on sampled/generated inputs — cutting inference cost. Copying the posterior over classes is called distillation. defined in ch. 12 — open in glossary replaces an expensive model — often a large regularized net or an ensemble — with a smaller one trained to mimic it. It works when the big model’s size was driven by preventing overfitting: once it has fit $f(x)$, you can generate unlimited $(x, f(x))$ training pairs to teach a compact student.

An alternative trains the student on the original points but to copy other features of the teacher — such as its posterior over the incorrect classes (distillation, Hinton 2015).

Next: computing only what each input needs, specialized hardware, and the start of computer vision — dynamic structure, hardware, and computer vision.