§1.2.1–1.2.2Three Waves of Neural Networks · Increasing Dataset Sizes

Deep learning looks brand-new, but it dates to the 1940s. It merely appears new because it was unpopular for years before its current rise — and because it has been rebranded many times, reflecting whichever researchers and perspectives held sway. Broadly there have been three waves: cybernetics cybernetics The first wave of neural-network research (1940s–60s): linear models of biological learning. defined in ch. 1 — open in glossary (1940s–60s), connectionism connectionism The second wave (1980s–90s): intelligent behavior from many networked simple computational units. defined in ch. 1 — open in glossary (1980s–90s), and the current resurgence under the name deep learning, beginning in 2006 (the book’s fig 1.7, retold below).

The neural perspective — and its limits

Some of the earliest learning algorithms were models of biological learning, hence the name artificial neural networks (ANNs). The neural perspective rests on two ideas: the brain is a proof by example that intelligent behavior is possible, so reverse-engineering it is one path to AI; and models that illuminate the brain are scientifically interesting in their own right. But the modern term “deep learning” appeals to a more general principle — learning multiple levels of composition — that needs no neural framing at all.

Wave 1: cybernetics and linear models

The earliest ancestors were simple linear models linear model A model based on f(x,w)=x·w; widely used but unable to learn XOR. defined in ch. 1 — open in glossary : take inputs $x_1, \dots, x_n$, learn weights $w_1, \dots, w_n$, output

The McCulloch-Pitts neuron (1943) recognized two categories by the sign of $f(\boldsymbol{x}, \boldsymbol{w})$ — but a human had to set the weights. The perceptron perceptron The first model that could learn its category-defining weights from labeled examples (Rosenblatt, 1958). defined in ch. 1 — open in glossary (Rosenblatt, 1958) was the first model to learn the weights from labeled examples. The ADALINE adaline Adaptive linear element: predicts real values with f(x); trained by a special case of stochastic gradient descent. defined in ch. 1 — open in glossary (Widrow & Hoff, 1960) returned $f(\boldsymbol{x})$ itself to predict real numbers — and its training rule was a special case of stochastic gradient descent stochastic gradient descent The dominant training algorithm: the gradient is an EXPECTATION, so estimate it on a small minibatch and step θ ← θ − εg — cost per update O(1) in dataset size. ADALINE's 1960 rule was a special case. defined in ch. 1 — open in glossary , whose slightly modified descendants still dominate deep-learning training today.

The XOR wall

XOR demands: $f([0,1]) = 1$ and $f([1,0]) = 1$, but $f([0,0]) = 0$ and $f([1,1]) = 0$. Try to find weights that do it:

Critics observing this flaw caused a backlash against biologically inspired learning — the first dip. When neural networks returned, XOR returned with them: §6.1 opens chapter 6 by solving exactly this problem with a tiny feedforward network.

What neuroscience still contributes

Today neuroscience is an inspiration, not the predominant guide — we simply cannot monitor thousands of interconnected neurons at once, so we do not understand even simple brain areas well enough to copy them. Its hints have been valuable nonetheless: ferrets whose visual signals are rewired to auditory cortex learn to “see” with it, suggesting one algorithm may serve most tasks; the many-simple-units-interacting idea is brain-inspired; the Neocognitron’s visual architecture became the modern convolutional network (§9.10); and today’s default model neuron, the rectified linear unit rectified linear unit g(z) = max{0, z}: the default hidden unit — near-linear, large consistent gradients, easy to optimize; dead where its activation is 0. (First introduced ch01.) defined in ch. 6 — open in glossary , simplifies a more brain-faithful ancestor (the Cognitron). But greater neural realism has not improved performance, and media claims that deep learning simulates the brain are wrong: it draws at least as much on linear algebra, probability, information theory and numerical optimization. Understanding the brain algorithmically is a separate, thriving field — computational neuroscience.

Wave 2: connectionism

Connectionism (parallel distributed processing) arose inside cognitive science in the 1980s, grounding models of cognition in neural implementations after symbolic models proved hard to map onto neurons — reviving Hebb’s 1940s ideas. Its central tenet: a large number of simple computational units achieve intelligent behavior when networked — true of biological neurons and hidden units alike. Three of its concepts remain load-bearing today:

1. Distributed representation distributed representation Each input represented by many features, each feature involved in many possible inputs. defined in ch. 1 — open in glossary — each input represented by many features, each feature involved in many inputs. Compare the two encodings of (car, truck, bird) × (red, green, blue):

2. Back-propagation back-propagation (Rumelhart 1986) the method that flows information from the cost backward to compute the gradient — a table-filling (dynamic-programming) application of the chain rule; NOT the whole learning algorithm. (First named ch01.) defined in ch. 6 — open in glossary — successfully used to train deep networks with internal representations and popularized in 1986; it has waxed and waned but is the dominant training approach as the book is written.
3. Sequence modeling advances (1990s) — Hochreiter (1991) and Bengio et al. (1994) diagnosed the fundamental difficulty of learning long dependencies (§10.7); Hochreiter & Schmidhuber answered in 1997 with the long short-term memory (LSTM) long short-term memory Gated recurrent network (1997) resolving the difficulty of learning long-term dependencies in sequences; still widely used. defined in ch. 1 — open in glossary , still widely used for sequence tasks including NLP at Google.

The wave ended in the mid-1990s: ventures over-promised, investors soured, and kernel machines and graphical models started winning benchmarks. Neural nets stayed quietly competitive on some tasks, kept alive by CIFAR’s NCAP initiative uniting Hinton (Toronto), Bengio (Montreal) and LeCun (NYU). Deep nets were “known” to be hard to train — we now know the 1980s algorithms worked all along; the hardware was just too slow to show it.

Wave 3 — and §1.2.2: why now? Data.

The 2006 breakthrough: Hinton trained a deep belief network efficiently by greedy layer-wise pretraining (§15.1); the other CIFAR groups quickly extended the strategy to many deep models, improving generalization and putting the depth in “deep learning.”

But the deeper answer to “why did it only work now?” is the data. The algorithms reaching human performance today are nearly identical to the ones struggling with toy problems in the 1980s. What changed is what we feed them. Society’s digitization keeps producing records; networked computers centralize them into datasets:

The age of “Big Data” lightened machine learning’s key burden — generalizing well from little data. The 2016 rules of thumb: a supervised deep model reaches acceptable performance with about 5,000 labeled examples per category, and matches or exceeds humans with 10 million+. Doing well below that threshold — by exploiting unlabeled data with unsupervised or semi-supervised learning — is a major open research area.