Increasing Model Sizes · Deep Learning Interactive

The other reason neural networks are wildly successful now after modest success since the 1980s: we finally have the computational resources to run large enough models. One of connectionism ’s main insights is that animals become intelligent when many of their neurons work together — a single neuron, or a small collection, is not particularly useful. This section measures “large” two ways: connections per neuron, and total neurons.

Connections per neuron: caught up decades ago

Biological neurons are not especially densely connected. Artificial networks have had a number of connections per neuron within an order of magnitude of mammalian brains for decades (fig 1.10): some networks match a cat, and matching a mouse is routine — even the human brain does not have an exorbitant number of connections per neuron. Today connection count is mostly a design choice, no longer a hardware limit.

Fig 1.10 (recreated) — 1 ADALINE · 2 Neocognitron · 3 GPU-accelerated convolutional network · 4 deep Boltzmann machine · 5 unsupervised convolutional network · 6 GPU MLP · 7 distributed autoencoder · 8 multi-GPU convolutional network · 9 COTS HPC network · 10 GoogLeNet.

Total neurons: astonishingly small until now

Total neuron count is a different story. Since the introduction of hidden units, artificial neural networks have doubled in size roughly every 2.4 years, driven by faster machines with more memory and by bigger datasets:

Term by term

N(t) = N_0 \cdot 2^{\,(t - t_0)/2.4}

$N(t)$	network size (number of neurons) in year t	count
$N_0$	size at the reference year t₀	count
$t - t_0$	years elapsed since the reference point	years
$2.4$	the observed doubling period — one more doubling every 2.4 years	years

What this really says

Growth is multiplicative, not additive: every 2.4 years the field doesn’t add a fixed number of neurons, it doubles whatever it already has. That means the gap between artificial nets and biological brains — several orders of magnitude — closes on a predictable schedule, but slowly: each ×10 takes about 8 years.

Worked example — when human-scale?

GoogLeNet-era networks (2014) have very roughly $10^7$ neurons; the human brain has ~ $10^{11}$ . The gap is $10^4$ , i.e. $\log_2 10^4 \approx 13.3$ doublings. At 2.4 years each: $2014 + 13.3 × 2.4 \approx 2046$ — and since biological neurons may compute more per unit than artificial ones, the book’s sober estimate is “not before the 2050s unless new technologies allow faster scaling.”

Drag the year and watch the trend cross the biological milestones:

Fig 1.11 (interactive extrapolation) — the 2.4-year doubling law vs. biology

year2014

projected size: 1.0×10^7 neurons(1.0×10^11 needed — on trend ≈2046)

Where we actually are on that ladder (fig 1.11): until quite recently networks were astonishingly small — smaller than a leech’s nervous system. The early perceptron and ADALINE had single neurons; the Neocognitron reached roughly a thousand; LeNet-5 (1998) sat near frog-egg-count scales; the 2010s GPU era pushed past the bee toward the frog. In retrospect it is not surprising that networks with fewer neurons than a leech could not solve sophisticated AI problems — and even today’s “large” networks are smaller than the nervous system of a frog.

Fig 1.11 (recreated) — 1 Perceptron · 2 ADALINE · 3 Neocognitron · 4 early back-propagation net · 5 RNN for speech · 6 MLP for speech · 7 mean-field sigmoid belief net · 8 LeNet-5 · 9 echo state net · 10 deep belief net · 11 GPU conv. net · 12 deep Boltzmann machine · 13 GPU DBN · 14 unsupervised conv. net · 15 GPU MLP · 16 OMP-1 · 17 distributed autoencoder · 18 multi-GPU conv. net · 19 COTS HPC · 20 GoogLeNet. Biological counts are approximate (Wikipedia, 2015); biological neurons may also represent more complicated functions, so the true gap may be larger than the plot portrays.

What drives the growth? Faster CPUs, the advent of general-purpose GPUs (the story of §12.1), faster network connectivity, and better distributed-computing software — one of the most important trends in deep learning’s history, and one generally expected to continue well into the future.

Check yourself — model sizes

1.On connections per neuron vs. total neurons — which statement matches the book?

2.In the extrapolation widget, roughly when does the 2.4-year doubling law carry a 2014 network (~10⁷ neurons) to human scale (~10¹¹)?

3.Which forces drive the growth in model size?

4.Why does the book caution that fig 1.11 may understate the gap between artificial and biological networks?

4 questions