Preprocessing · Contrast Normalization · Speech Recognition

§12.2.1 Preprocessing

Vision needs surprisingly little preprocessing. The one strict requirement is a consistent pixel range — mixing images in $[0,1]$ with images in $[0,255]$ fails. Resizing to a standard size is common but not always necessary (some conv nets accept variable sizes). Dataset augmentation (§7.4) — random translations, rotations, flips, color perturbation — is one of the best ways to cut vision generalization error; a test-time cousin averages predictions over several crops (an ensemble). Other preprocessing puts examples into a more canonical form to reduce variation the model must handle — but with large data and models, it’s often best to just let the model learn its own invariances (AlexNet’s only preprocessing was subtracting the per-pixel mean).

§12.2.1.1 Contrast normalization

A safe-to-remove source of variation is contrast — the standard deviation of an image’s pixels (eqs 12.1–12.2). Global contrast normalization (GCN) subtracts the mean and rescales every image to a fixed pixel std $s$ :

Global contrast normalization (eq 12.3)

X'_{i,j,k} = s\,\frac{X_{i,j,k} - \bar X}{\max\!\Big(\epsilon,\ \sqrt{\lambda + \tfrac{1}{3rc}\sum_{i,j,k}\big(X_{i,j,k}-\bar X\big)^2}\Big)}

$X_{i,j,k} - \bar X$	subtract the mean image intensity — center the image so its average is zero	centering
$\sqrt{\lambda + \text{std}^2}$	the (regularized) contrast = pixel standard deviation; λ biases the estimate so near-constant images are not amplified	the contrast
$\max(\epsilon,\ \cdot)$	a floor to avoid dividing by zero on a zero-contrast image	safety
$s$	the target standard deviation (often 1) — every image ends with the same contrast	scale

Geometrically, GCN maps examples onto a spherical shell (fig 12.1) — useful because networks respond better to directions than exact locations. Slide $\lambda$ to see the effect:

Fig 12.1 (interactive) — GCN maps 2-D examples toward a sphere. λ=0 puts every non-zero example exactly on the sphere (discarding all norm variation); λ>0 draws them toward it while keeping some, avoiding noise amplification in low-contrast images.

regularizer λ = 0.00

● raw● after GCN

λ = 0: every non-zero example is mapped exactly onto the sphere — GCN discards all norm (contrast) variation, keeping only direction. Networks respond better to directions than exact locations.

(Do not confuse this with sphering / whitening , which instead rescales the principal components to equal variance.) GCN keeps a big dark region distinguishable from a big bright one, but won’t make edges within the dark region pop. Local contrast normalization (LCN) fixes that by normalizing each small window (fig 12.2) — subtract a local mean, divide by a local std — making edges stand out. LCN is efficiently computed via separable convolution (§9.8), is differentiable (usable as a layer, not just preprocessing), and needs more regularization than GCN because small windows often have near-zero std.

§12.3 Speech recognition

Automatic speech recognition (ASR) maps an acoustic signal to the intended words. With $X=(x^{(1)},\dots,x^{(T)})$ the acoustic frames (traditionally 20 ms each):

The ASR objective (eq 12.4)

f^{*}_{\text{ASR}}(X) = \arg\max_{y}\ P^{*}\!\big(y \mid X = X\big)

$X = (x^{(1)},\dots,x^{(T)})$	the acoustic input sequence — one vector per ~20 ms audio frame	input
$y = (y_1,\dots,y_N)$	the target linguistic sequence (words or characters)	output
$\arg\max_y P^{*}(y\mid X)$	return the most probable word/character sequence given the audio	decode

For decades GMM-HMM systems ruled (an HMM over phoneme sequences, a GMM turning each symbol into audio). Deep learning changed that fast:

TIMIT phoneme-error-rate progression as speech recognition shifted to deep learning.

Starting in 2009, deep feedforward nets — each layer initialized by unsupervised RBM pretraining — replaced the GMM acoustic model, cutting TIMIT phoneme error from ~26% to 20.7%. Then ReLU + dropout made pretraining unnecessary, and ~30% word-error-rate gains flipped the whole industry to deep learning within two years. Two later pushes: 2-D convolutions treating the spectrogram as an image (time × frequency), and end-to-end systems that remove the HMM entirely — a deep LSTM RNN (§10.10) trained with CTC brought TIMIT to a record 17.7%, with depth from both stacked layers and time unfolding.

Next: the largest applications area — natural language processing, from n-grams to word embeddings — NLP, n-grams, and neural language models.

Check yourself — vision preprocessing and speech recognition

1.What preprocessing does computer vision strictly require?

2.What is global contrast normalization (GCN), and what does the λ regularizer do? (See the ContrastNormLab.)

3.How does local contrast normalization (LCN) differ from GCN?

4.Why is dataset augmentation especially effective for object recognition?

5.What model family dominated speech recognition before ~2012, and how did deep learning first improve it?

6.What characterizes end-to-end deep-learning speech recognition?

6 questions