At the same time that scale grew, deep networks got more accurate and took on harder tasks — the fourth and final historical trend.
Object recognition: the ILSVRC story
The earliest deep models recognized single objects in tightly cropped, extremely small images. Modern networks process rich high-resolution photographs without cropping, and recognize at least 1,000 object categories instead of two. The largest contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), provided the field’s dramatic moment — step through the book’s fig 1.12:
1Before deep learning (2010–2011)
State-of-the-art top-5 error hovers around 26–28%: a model may output five guesses per image, and for more than a quarter of test images none of the five is right.
The same trajectory played out elsewhere: after speech-recognition error rates stagnated from about 2000, deep learning’s arrival produced a sudden drop — some error rates cut in half (§12.3 tells the full story). Deep nets also achieved spectacular pedestrian detection and image segmentation, and superhuman traffic-sign classification.
From labels to programs: rising task complexity
Accuracy on fixed tasks is only half the trend; the kind of task grew too — from emitting one label toward emitting structure, and finally toward learning behavior:
The complexity ladder (diagram not in the book) — each rung was widely considered out of reach shortly before it fell.
| Breakthrough | Work | |
|---|---|---|
| Sequence transcription | Image → whole character sequence, no per-character labels | Goodfellow et al., 2014 |
| Sequence-to-sequence learning | Sequences → sequences (machine translation) | Sutskever 2014 · Bahdanau 2015 |
| Neural Turing machines | Learn simple programs from examples | Graves et al., 2014 |
| Deep reinforcement learning | Human-level Atari play from trial and error | DeepMind (Mnih et al.), 2015 |
The last row is reinforcement learning reinforcement learning Learning to perform a task by trial and error, from rewards rather than labeled examples, without guidance from a human operator. defined in ch. 1 — open in glossary : an autonomous agent learning a task by trial and error, with no guidance from a human operator — deep learning’s extension into behavior.
Impact — commercial and scientific
These applications are highly profitable: deep learning is used by Google, Microsoft, Facebook, IBM, Baidu, Apple, Adobe, Netflix, NVIDIA and NEC. The advances lean on software infrastructure as much as hardware — Theano, PyLearn2, Torch, DistBelief, Caffe, MXNet and TensorFlow each supported important research or products.
Deep learning has also given back to science: modern convolutional networks serve neuroscientists as a working model of visual processing; deep models predict how molecules interact (helping design drugs), search for subatomic particles, and parse microscope images to assemble 3-D maps of the human brain — with more scientific fields expected to follow.
In summary: deep learning is an approach to machine learning that drew on decades of knowledge about brains, statistics and applied math, and has grown tremendously in usefulness thanks to more powerful computers, larger datasets, and techniques for training deeper networks. The years ahead — this book’s subject — are full of challenges and opportunities to take it further. That completes chapter 1; the technical journey begins with the mathematics: chapter 2, Linear Algebra.
Check yourself — accuracy, complexity, impact
1.In the ILSVRC widget, what happened in 2012?
2.What does 'top-5 error = 15.3%' actually mean?
3.Why was the 2014 sequence-transcription result (reading a whole character string from an image) notable?
4.What distinguishes reinforcement learning from the supervised successes earlier in the section?
5.Which is an example of deep learning contributing back to other sciences?