Motivation — Sparse Interactions, Parameter Sharing, Equivariance

§9.2 Why convolution helps

Convolution leverages three ideas that improve a machine-learning system — sparse interactions, parameter sharing, and equivariant representations — plus a bonus: it handles variable-size inputs.

Sparse interactions

A dense layer connects every output to every input. Convolution instead uses sparse interactions : the kernel is much smaller than the input, so each output depends on only a few inputs. You can detect an edge with a kernel spanning tens of pixels even though the image has millions:

Term by term

\underbrace{m \times n}_{\text{dense}} \;\longrightarrow\; \underbrace{k \times n}_{\text{sparse}} \;\xrightarrow{\text{sharing}}\; \underbrace{k}_{\text{shared}}

$m \times n$	a fully-connected layer stores a weight for EVERY (input, output) pair — with millions of pixels this is enormous, and runtime is O(m×n)	dense
$k \times n$	limit each output to k inputs (the kernel width) → only k×n params and O(k×n) runtime, with k orders of magnitude smaller than m	sparse
$k$	parameter sharing (next) reuses the SAME kernel everywhere, so storage drops to just k weights (runtime stays O(k×n))	shared

Each output has a small receptive field — the inputs that affect it. But stack layers, and a deep unit’s receptive field grows, so it can indirectly see most of the input despite every connection being sparse (figs 9.2–9.4):

Parameter sharing (or tied weights ) means the same kernel weight is used at every position, rather than a separate weight per location. This does not change the forward runtime ( $O(k\times n)$ ) but drops storage to just $k$ weights. The efficiency is staggering: the book’s vertical-edge example needs ~268,000 operations by convolution versus over 8 billion matrix entries for the same transformation — roughly 60,000× cheaper to compute. Move the cat and watch one shared detector find it at any location:

Fig 9.5 — parameter sharing: one shared kernel is applied at every location, so the cat is detected wherever it appears with a single set of weights

cat @ col 2

shared: 2 weights (one detector)

un-shared: 14 weights (7 × 2)

saving grows with image size

Move the cat: the same detector fires wherever it appears — so the model needs to learn the "cat" pattern only once, not separately at every column. That is parameter sharing: force the weights equal across positions, store just one copy.

Equivariance to translation

Parameter sharing gives convolution a special property: equivariance to translation .

Term by term

f(g(\boldsymbol{x})) = g(f(\boldsymbol{x}))

$g$	a shift (translation) of the input — move the object/event to a new position	the shift
$f$	the convolution. Shifting THEN convolving equals convolving THEN shifting — so the feature map moves exactly as the input moved	the convolution
$\text{not scale/rotation}$	convolution is equivariant to translation only — it is NOT equivariant to scaling or rotating the image, which need other mechanisms	the limit

Slide the feature and watch its representation slide by the identical amount:

Translation equivariance: move the edge feature and the peak in the feature map moves the same distance — detect a feature once, find it anywhere

feature at col 2

feature at column 2

→ peak response at column 2

shift the input, the map shifts identically

Because the same kernel is applied everywhere, moving the feature moves its representation by the same amount — f(g(x)) = g(f(x)), equivariance to translation. (Convolution is not equivariant to scale or rotation — those need other mechanisms.)

Sharing is not always wanted — for face-centered images you might want different features at different locations (eyebrows on top, chin on the bottom). And convolution’s variable-size handling (a kernel applied a different number of times) is a bonus we return to in §9.7.

Next: pooling, and the view of convolution and pooling as an infinitely strong prior — pooling & the infinitely strong prior.

Check yourself — sparse interactions, parameter sharing, equivariance

1.How do sparse interactions make convolution more efficient than a dense layer?

2.In the receptive-field diagram, why does the deep unit g₃ 'see' all five inputs even though each connection is sparse?

3.What does parameter sharing (tied weights) do, and what is its cost/benefit?

4.In EquivarianceLab, moving the edge feature moves the peak response by the same amount. What property is this, and how is it defined?

5.Which transformations is convolution NOT naturally equivariant to, and when might you NOT want parameter sharing?

5 questions