§7.8–7.9Early Stopping · Parameter Tying and Parameter Sharing

Part II DL pp. 246–253 · ~7 min read

  • early stopping
  • patience
  • effective capacity (of training time)
  • parameter tying
  • parameter sharing

§7.8 Early stopping

Train a model with enough capacity to overfit and you see it reliably: the training loss keeps falling, but the validation loss bottoms out and starts climbing again. The fix is almost embarrassingly simple — keep a copy of the parameters every time validation improves, and when it has not improved for a while, return that saved copy instead of the latest one. That is early stopping , probably the most-used regularizer in deep learning. Sweep the epoch and watch the return point:

Fig 7.3 (interactive) — training loss falls monotonically; validation traces an asymmetric U. We return the params at the lowest-validation epoch, and stop after “patience” worsening checks
epochslossreturntrainingvalidation
best val @ epoch 60 = 0.124
current val = 0.124
epochs since best = 0
still training… (30 to go)

Training loss keeps falling, but we return the parameters from the lowest-validation epoch, not the last one. A single run tries every "number of steps" — that is why early stopping is such a cheap hyperparameter search.

The bookkeeping is one loop:

Algorithm 7.1 — early stopping (click lines)
  1. 1Let nn = steps between evaluations, pp = patience, θo\boldsymbol{\theta}_o = initial parametersⓘ
  2. 2θ←θo; i←0; j←0; v←∞; θ∗←θ; i∗←i\boldsymbol{\theta}\leftarrow\boldsymbol{\theta}_o;\ i\leftarrow 0;\ j\leftarrow 0;\ v\leftarrow\infty;\ \boldsymbol{\theta}^*\leftarrow\boldsymbol{\theta};\ i^*\leftarrow iⓘ
  3. 3while j<pj < p doⓘ
  4. 4train nn steps;  i←i+n; v′←ValidationError(θ)\ i\leftarrow i+n;\ v'\leftarrow \mathrm{ValidationError}(\boldsymbol{\theta})ⓘ
  5. 5if v′<vv' < v: j←0; θ∗←θ; i∗←i; v←v′j\leftarrow 0;\ \boldsymbol{\theta}^*\leftarrow\boldsymbol{\theta};\ i^*\leftarrow i;\ v\leftarrow v'ⓘ
  6. 6else j←j+1j\leftarrow j+1ⓘ
  7. 7return best parameters θ∗\boldsymbol{\theta}^*, best step count i∗i^*ⓘ

Early stopping is free hyperparameter search

“Number of training steps” is itself a hyperparameter with a U-shaped validation curve — but unlike weight decay’s α\alpha, a single training run tries every value of it at once. The only real cost is evaluating the validation set periodically (run it in parallel, or less often) and keeping one spare copy of the parameters. It changes nothing about the objective, the parameters, or the learning dynamics — so it composes with any other regularizer, and it will not trap the network in a bad small-weight minimum the way heavy weight decay can.

Early stopping holds out a validation set, so afterward you can reclaim that data with a second training round. Two strategies:

Reclaiming the validation data after early stopping
what it doescaveat
Alg 7.2 — retrain from scratchReinitialize, train on ALL data for the i* steps early stopping found.Unclear whether to match the same number of parameter updates or the same number of passes — the bigger set needs more updates per pass.
Alg 7.3 — continue trainingKeep the first-round parameters, keep training on all data until validation loss falls below the training objective value where stopping halted.Cheaper (no restart) but not well-behaved — the target may never be reached, so it is not even guaranteed to terminate.
Dotted-underlined cells have explanations — click one.

Why it regularizes — early stopping ≈ L²

Bishop and others showed early stopping restricts optimization to a small volume of parameter space around the start θo\boldsymbol{\theta}_o (fig 7.4): the product of steps τ\tau and learning rate ϵ\epsilon is an effective capacity that bounds how far the parameters can travel. Compare the two pictures — a trajectory stopped early lands in the same place weight decay pulls to:

early stoppingθₒw̃w*L² regularization0w̃w*

For a linear model with a quadratic cost, this equivalence is exact:

Early stopping equals L² regularization (equations 7.33–7.45)
J^(w)=J(w∗)+12(w−w∗)⊤H(w−w∗)\hat{J}(\boldsymbol{w}) = J(\boldsymbol{w}^*) + \tfrac12 (\boldsymbol{w}-\boldsymbol{w}^*)^\top \boldsymbol{H} (\boldsymbol{w}-\boldsymbol{w}^*)

step 1/6: Quadratic approximation of the cost around the optimum w*, with H the positive-semidefinite Hessian there.

The advantage over weight decay: early stopping finds the right amount of regularization automatically, in one run, while weight decay needs many experiments over different α\alpha.

§7.9 Parameter tying and parameter sharing

Weight decay pulls parameters toward a fixed point (zero). But sometimes our prior knowledge is about relationships between parameters — that two of them should be close. Say models AA and BB solve the same task on slightly different input distributions and we believe wi(A)≈wi(B)w_i^{(A)}\approx w_i^{(B)}. Express that with a tying penalty:

Term by term

Ω(w(A),w(B))=∥w(A)−w(B)∥22\Omega(\boldsymbol{w}^{(A)}, \boldsymbol{w}^{(B)}) = \lVert \boldsymbol{w}^{(A)} - \boldsymbol{w}^{(B)} \rVert_2^2
w(A),w(B)\boldsymbol{w}^{(A)}, \boldsymbol{w}^{(B)}the parameters of two related models (e.g. a supervised classifier and an unsupervised model of the same inputs)two models
∥⋅∥22\lVert \cdot \rVert_2^2penalize the DIFFERENCE, pulling the two parameter sets toward each other rather than toward zerotying penalty
vs sharing\text{vs sharing}the stronger, more popular option: FORCE the sets equal (w^(A) = w^(B)) so only ONE copy must be storedthe upgrade

The stronger cousin is parameter sharing : force sets of parameters to be equal. Besides the regularizing constraint, it saves memory — only the unique set is stored. Its flagship is the convolutional network: natural images are translation-invariant (a cat shifted one pixel is still a cat), so a CNN applies the same feature detector at every location. Move the cat and watch one shared detector find it anywhere:

Parameter sharing in a CNN: one shared 2-cell “cat detector” is applied at every position, firing wherever the cat is — one set of weights, not one per location
input image (a “cat” = two lit cells)🐱🐱shared detectordetector output at each position✓
shared: 2 weights (one detector)
un-shared: 14 weights (7 × 2)
saving grows with image size

Move the cat: the same detector fires wherever it appears — so the model needs to learn the "cat" pattern only once, not separately at every column. That is parameter sharing: force the weights equal across positions, store just one copy.

Sharing lets CNNs slash their unique parameter count and grow much larger without needing more training data — the best example of baking domain knowledge straight into the architecture (much more in chapter 9).

Next: sparse representations and the ensemble methods (bagging) that dropout will build on — sparse representations & bagging.

Check yourself — early stopping and parameter sharing

1.What does early stopping actually return, and when does it stop?

2.In EarlyStoppingLab, you slide to epoch 200 with patience 30. Training loss is still dropping but the 'return' marker sits back near epoch 87. Why?

3.In what precise sense is early stopping equivalent to L² regularization?

4.What is the difference between parameter tying and parameter sharing?

5.In ParamShareLab, moving the cat keeps the SAME detector firing at its new location. What two benefits does this parameter sharing give a CNN?

5 questions