The Neuroscientific Basis for Convolutional Networks

§9.10 What the brain taught the convolutional network

Convolutional networks are perhaps the greatest success story of biologically inspired AI. The story starts with Hubel and Wiesel’s Nobel-winning cat experiments: neurons in the early visual system fire strongly for very specific patterns (precisely oriented bars) and hardly at all for others. Light flows from the retina through the LGN to V1 , the primary visual cortex, then deeper — and this pathway inspired the whole architecture:

A conv layer captures three properties of V1: it is a spatial map (2-D features mirroring the image); it has simple cells — roughly linear in a small receptive field, the model for a CNN’s detector units; and complex cells — like simple cells but invariant to small shifts, the model for pooling units. Repeat detection→pooling deeper and you eventually reach cells that respond to a whole concept invariant to many transformations — grandmother cells . These really exist: the “Halle Berry neuron” in the medial temporal lobe fires for a photo, a drawing, or the text “Halle Berry.” The closest analog to a CNN’s last feature layer is the inferotemporal cortex (IT) , whose first-~100ms feedforward firing rates a CNN can actually predict.

What the cells detect: Gabor functions

We can’t read a neuron’s weights directly, so neuroscientists show white-noise images and fit a linear model to its responses — reverse correlation . It reveals that most V1 simple-cell weights are Gabor functions : a Gaussian gate times an oriented cosine grating. Adjust the parameters:

Fig 9.18 — a Gabor filter: a Gaussian gate (width β) × an oriented cosine grating (frequency f, phase φ, angle τ). Amber = positive weight, blue = negative

positive weight negative weight

freq f = 1.6phase φ = 0.0angle τ = 0.4width β = 0.18

A Gaussian gate (width β) times an oriented cosine grating (frequency f, phase φ, angle τ). This is what reverse correlation recovers from V1 simple cells — and what the first layer of almost every vision model learns.

Term by term

w(x,y) = \underbrace{\alpha\, e^{-\beta_x x'^2 - \beta_y y'^2}}_{\text{Gaussian gate}} \cdot \underbrace{\cos(f x' + \phi)}_{\text{oriented grating}}

$\alpha\, e^{-\beta x'^2 - \beta y'^2}$	the Gaussian GATE: restricts the response to near the receptive-field center (α sets magnitude, β the falloff)	where
$\cos(f x' + \phi)$	an oriented brightness WAVE: f is the spatial frequency, φ the phase. The cell fires most when the image's brightness wave is IN PHASE with the weights	what
$x', y'\ (\text{rotated})$	x₀,y₀,τ translate and rotate the frame — the cell responds along a line rotated τ from horizontal, centered at (x₀,y₀)	orientation

Complex cells and quadrature pairs

A simple cell responds to a specific spatial frequency, in a specific direction, at a specific location — and is phase-sensitive. A complex cell takes the L² norm of two simple cells whose only difference is a quarter-cycle phase shift — a quadrature pair — making it respond to that grating regardless of phase, i.e. invariant to small translations (and to negating the image). This is exactly the detector→pooling motif, written in biology. Strikingly, sparse coding and almost every deep-learning algorithm learn Gabor-like edge detectors in their first layer — so learning the “right” filters is not, by itself, evidence of a correct brain model.

That said, CNNs and real vision differ in many ways: the eye is low-resolution outside the tiny fovea and uses saccades + attention to stitch a scene (attention is now an active research direction); biological vision integrates other senses and heavy top-down feedback; and a real neuron is probably not a single linear filter (a modern V1 model uses several quadratic filters per cell, blurring the simple/complex distinction into a continuum). Neuroscience also said little about training: parameter sharing predates back-prop (Fukushima’s Neocognitron, 1980), which learned by unsupervised clustering; back-propagation entered via TDNNs (1-D CNNs, Lang & Hinton 1988) and then LeCun’s 2-D image CNNs.

Next: convolutional networks and the history of deep learning — the history.

Check yourself — the neuroscience of convolutional networks

1.Which two V1 cell types inspired which parts of a convolutional layer?

2.What is a 'grandmother cell', and what is the closest brain analog to a CNN's last feature layer?

3.A Gabor function is a Gaussian gate times an oriented cosine grating. In GaborLab, what do f, φ, and τ control?

4.How does a complex cell achieve invariance, and what is a quadrature pair?

5.Many learning algorithms learn Gabor-like first-layer filters. What does the book conclude from this?

5 questions