§7.0–7.1.1Regularization: The Bias–Variance Trade · Parameter Norm Penalties · L² Weight Decay

§7.0 What regularization is really trading

Chapter 6 gave us powerful function approximators. Now the problem flips: a model with enough capacity to fit the training set will also fit its noise. Regularization regularization Any modification intended to reduce generalization error but not training error — preferences among hypothesis-space members (e.g. weight decay λwᵀw). defined in ch. 5 — open in glossary is “any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.” Almost every strategy in this chapter buys lower test error at the price of higher training error — and that is exactly the point.

The mechanism is the bias–variance trade bias-variance trade-off MSE = Bias² + Variance: capacity ↑ typically bias ↓ variance ↑ — the U-curve again; negotiated by cross-validation. defined in ch. 5 — open in glossary : regularizing an estimator increases bias to reduce variance. An effective regularizer makes a profitable trade — cutting variance a lot while adding only a little bias. Chapter 5 framed three regimes: the model family (1) excludes the true process (underfitting, high bias), (2) matches it, or (3) contains it plus many other processes (overfitting, high variance). Regularization aims to move a model from regime 3 into regime 2.

§7.1 Parameter norm penalties

The oldest and simplest family adds a parameter norm penalty parameter norm penalty A term αΩ(θ) added to the objective (J̃ = J + αΩ) that limits capacity by penalizing weight size; α∈[0,∞) sets the strength. Only weights are penalized, not biases. defined in ch. 7 — open in glossary $\Omega(\boldsymbol{\theta})$ to the objective:

§7.1.1 L² regularization (weight decay)

The most common penalty is the L² norm, $\Omega(\boldsymbol{\theta}) = \tfrac12\lVert\boldsymbol{w}\rVert_2^2$ — L² regularization l2 regularization Weight decay: Ω = ½‖w‖₂². Shrinks each Hessian-eigencomponent of w* by λᵢ/(λᵢ+α), decaying low-curvature (unimportant) directions while preserving high-curvature ones. defined in ch. 7 — open in glossary , better known as weight decay weight decay The L² regularizer: J = MSE + λwᵀw; trades data fit against small weights; = MAP with a Gaussian prior; = ch4's constrained optimization. defined in ch. 5 — open in glossary (and, elsewhere, ridge regression or Tikhonov regularization ridge regression / tikhonov regularization Other-community names for L² parameter regularization (weight decay). defined in ch. 7 — open in glossary ). Look at what it does to a single gradient step (assume $\boldsymbol{\theta}=\boldsymbol{w}$):

That is one step. To see the effect over all of training, approximate $J$ by a quadratic bowl around the unregularized optimum $\boldsymbol{w}^*$ and solve for where weight decay moves the minimum:

The picture: weight decay rescales $\boldsymbol{w}^*$ along the eigenvectors of the Hessian. Directions the objective cares about (large $\lambda_i$) barely move; directions it is indifferent to (small $\lambda_i$) get shrunk to nearly zero. Drag $\alpha$, $\lambda_1$, $\lambda_2$ and watch the equilibrium $\tilde{\boldsymbol{w}}$ slide between the objective’s optimum and the origin:

The linear-regression special case

For linear regression the cost is truly quadratic, so the analysis is exact. Weight decay changes the normal equations by adding $\alpha$ to the diagonal:

Next: L¹ regularization (which makes weights exactly zero), norm penalties as constrained optimization, and dataset augmentation — L¹, constraints & augmentation.