Pooling · Convolution and Pooling as an Infinitely Strong Prior

§9.3 Pooling

A typical convolutional layer has three stages (fig 9.7): a convolution stage (parallel affine convolutions → linear activations), a detector stage (a nonlinearity, e.g. ReLU), and a pooling stage:

A pooling function replaces an output with a summary statistic of its neighborhood — most commonly max pooling (the maximum in a rectangular region; also average, L², or distance-weighted). The payoff is approximate invariance to small translations : shift the input a little and most pooled outputs do not change (fig 9.8). Toggle the shift and adjust the stride:

Max pooling: shifting the input by one pixel leaves most pooled outputs unchanged (invariance); a larger stride downsamples, giving the next layer fewer inputs

detector stage

0.1

1.0

0.2

1.0

0.3

0.1

1.0

0.2

0.4

max pool (width 3, stride 1) — 7 units

1.0

shift the input by one pixel

stride

after a 1-pixel shift:

0 of 7 pooled outputs changed

stride 2 → 4 units (downsampled)

Max pooling reports only the largest value in each window, so shifting the input changes most pooled outputs not at all — approximate invariance to small translations. Increasing the stride also downsamples, handing the next layer fewer inputs.

Invariance is valuable when you care whether a feature is present, not exactly where — to detect a face you need an eye on the left and one on the right, not their pixel-perfect locations. (When precise location matters, e.g. finding where two edges meet, pooling can hurt.) Two more facts: pooling over separately-parametrized convolutions lets the features learn which transformations to be invariant to (fig 9.9 — three filters for differently- rotated 5s, max-pooled → rotation invariance, the maxout idea); and by spacing pooling regions $k$ pixels apart (downsampling, fig 9.10) you also shrink the representation, which is essential for feeding a fixed-size classifier from variable-size inputs.

§9.4 Convolution and pooling as an infinitely strong prior

Recall a prior (§5.2): beliefs about reasonable models before seeing data. Priors range in strength:

Term by term

\text{weak (high entropy)} \;\longrightarrow\; \text{strong (low entropy)} \;\longrightarrow\; \text{infinitely strong}

$\text{weak}$	a high-entropy prior (e.g. a high-variance Gaussian) that lets the data move the parameters more or less freely	lenient
$\text{infinitely strong}$	places ZERO probability on some parameters — forbidding those values entirely, regardless of what the data say	forbidding
$\text{conv \& pool}$	convolution = an infinitely strong prior (a layer's weights are zero outside a small receptive field, and shared/shifted across space — local + translation-equivariant); pooling = each unit must be invariant to small translations	the CNN prior

So a conv net is like a fully-connected net with an infinitely strong prior baked into its weights. That view yields two insights:

Two consequences of the prior

1. It can cause underfitting. Like any prior, convolution and pooling only help when their assumptions are right. If a task needs precise spatial information, pooling every feature raises the training error — some architectures therefore pool only some channels. When a task couples very distant input locations, the locality of convolution is inappropriate.
2. Only compare conv models to conv models. A model without convolution could still learn even if you permuted all the pixels — it has no built-in notion of topology. So image benchmarks separate permutation-invariant models (which must discover spatial structure) from models with spatial knowledge hard-coded in.

Next: the variants of convolution actually used in practice — multi-channel, strided, and zero-padded — convolution variants, part 1.

Check yourself — pooling and the infinitely strong prior

1.What are the three stages of a typical convolutional layer?

2.In PoolingLab, toggling 'shift the input by one pixel' changes only a few pooled outputs. Why?

3.Besides invariance, what does increasing the pooling stride accomplish, and why is pooling essential for variable-size inputs?

4.In what sense are convolution and pooling an 'infinitely strong prior'?

5.What two insights follow from viewing convolution/pooling as an infinitely strong prior?

5 questions