Convolution Variants I — Multi-Channel, Strided, Zero-Padding

§9.5 The convolution used in practice

The convolution in a neural network is not quite the textbook operation. Two differences matter right away. First, a layer runs many convolutions in parallel — one kernel extracts one feature, and we want many features. Second, the input is usually multi-channel (a color image has R, G, B at each pixel; a hidden layer’s input is the many feature maps of the layer below). So we think in 3-D tensors (one channel axis + two spatial axes), and software adds a fourth (batch) axis.

Multi-channel convolution

Multi-channel convolution uses a 4-D kernel $K_{i,l,m,n}$ and sums over input channels:

Term by term

Z_{i,j,k} = \sum_{l,m,n} V_{l,\, j+m-1,\, k+n-1}\, K_{i,l,m,n}

$K_{i,l,m,n}$	the 4-D kernel: output channel i, INPUT channel l, and spatial offsets (m,n)	the weights
$V_{l,\,\cdot,\,\cdot}$	the input value in input-channel l at the offset spatial position	the input
$\sum_l$	crucially, each output channel i sums over ALL input channels l — that mixing is how a layer builds new features from the previous layer's maps	channel mixing

Strided convolution

To cut cost, sample the output only every $s$ pixels — a strided convolution (fig 9.12):

Term by term

Z_{i,j,k} = c(\boldsymbol{K}, \boldsymbol{V}, s)_{i,j,k} = \sum_{l,m,n} V_{l,\, (j-1)s+m,\, (k-1)s+n}\, K_{i,l,m,n}

$(j-1)s + m$	consecutive output positions are s pixels apart in the input — the output is downsampled by a factor of s	the stride
$s$	the stride: larger s = fewer output cells and less computation (a separate stride per axis is possible)	skip factor
$\equiv \text{conv + downsample}$	mathematically the same as unit-stride convolution followed by downsampling — but computed directly, without the wasted work	the shortcut

Zero-padding: valid, same, full

The other essential knob is zero-padding — implicitly surrounding the input with zeros. Without it, every convolution shrinks the width by $k-1$ , so you must choose between shrinking fast or using tiny kernels. Padding decouples the output size from the kernel width. Slide the input width, kernel width, stride, and padding mode and watch the output size:

Output size = ⌊(m + pad − k)/s⌋ + 1. Valid (no pad) shrinks to m−k+1; same preserves m; full grows to m+k−1; stride downsamples

m = 9k = 3stride = 1

same: output = m (stride 1)

= ⌊(9 + 2 − 3)/1⌋ + 1 = 9

valid shrinks the map (m−k+1) so depth is limited; same pads to preserve size for arbitrarily deep stacks; full grows it to m+k−1. The stride downsamples on top. Optimal padding usually sits between valid and same.

The three named cases: valid uses no padding, so the output shrinks to $m-k+1$ each layer (limiting depth); same pads just enough to keep the output equal to the input, allowing arbitrarily deep stacks (though border pixels influence fewer outputs); and full pads enough that every input pixel is visited $k$ times, growing the output to $m+k-1$ (its border outputs depend on fewer inputs). In practice the optimal padding usually lies between valid and same.

Next: locally connected and tiled convolution, transpose convolution, and structured outputs — convolution variants, part 2 & structured outputs.

Check yourself — multi-channel, strided, and zero-padded convolution

1.Why does a practical convolutional layer use a 4-D kernel and sum over input channels?

2.What does a strided convolution do, and how does it relate to unit-stride convolution?

3.In StrideLab, with m=9, k=3, stride=1 and 'valid' padding, what is the output size, and why does valid convolution limit network depth?

4.What does zero-padding accomplish, and what is the trade-off of 'same' vs 'full'?

5.Without zero-padding, what forces an undesirable choice, and what does padding avoid?

5 questions