Regularization · Hyperparameters & Validation Sets · Point Estimation

§5.2.2 Regularization

No free lunch says design for specific tasks — by building preferences into the learner. Changing the hypothesis space (degree) was the blunt version; the refined version keeps all functions eligible but prefers some: the unpreferred win only by fitting significantly better. The flagship: weight decay for linear regression —

Term by term

J(\boldsymbol{w}) = \mathrm{MSE}_{\mathrm{train}} + \lambda\, \boldsymbol{w}^\top\boldsymbol{w}

$\mathrm{MSE}_{\mathrm{train}}$	fit the data	data term
$\boldsymbol{w}^\top\boldsymbol{w}$	the REGULARIZER Ω(w): prefer small squared L² norm — smaller slopes, weight on fewer features	preference term
$\lambda$	preference strength, chosen ahead of time: λ = 0 imposes nothing; large λ forces weights toward zero — this is ch. 4’s constrained optimization wearing its penalty coat	scalar ≥ 0

Fig 5.5 live — the same degree-9 model as last section, tamed and un-tamed by λ:

Fig 5.5 (interactive) — degree 9 throughout; only λ changes. Huge λ → flat underfit; medium → the right shape; λ → 0 → the overfit of fig 5.2

degree9log₁₀λ-8show test points

train 0.018 · test 50.985

overfitting

The definition worth memorizing

Regularization is any modification made to a learning algorithm that is intended to reduce its generalization error but not its training error. Excluding functions = an infinitely strong preference against them; penalties are the graded version. It rivals optimization as the field’s central concern — and (NFL again) there is no best form, only forms suited to tasks. Deep learning’s bet: very general-purpose regularizers can serve the whole range of intellectual tasks. Chapter 7 is this idea grown up.

§5.3 Hyperparameters and validation sets

Settings like the polynomial degree and λ are hyperparameters : values the learning algorithm does not itself adapt. Sometimes because they’re hard to optimize — more often because learning them on the training set would always choose maximum capacity (higher degree, λ = 0: guaranteed overfitting). They need data the trainer never observes: the validation set , split off from the training data (typically 80 % train / 20 % validation) — never from the test set, which must make no choices about the model:

Since the validation set “trains” the hyperparameters, its error underestimates generalization error too (less than training error does); the final estimate comes from the untouched test set.

Aside — even test sets go stale

When a community benchmarks against the same test set for years — counting every attempt to beat the state of the art — evaluations become optimistic there as well. Benchmarks age; the field moves on to newer, larger, more ambitious datasets.

§5.3.1 Cross-validation

A small test set means high statistical uncertainty in the error estimate — hard to claim algorithm A beats B. When data is scarce, k-fold cross-validation uses all of it, at k× the compute:

Algorithm 5.1 — KFoldXV(D, A, L, k) (click lines)

1split $D$ into $k$ mutually exclusive subsets $D_i$ , union $= D$ ⓘ
2for $i$ from 1 to $k$ doⓘ
3 $f_i = A(D \setminus D_i)$ ⓘ
4for $z^{(j)}$ in $D_i$ do $e_j = L(f_i, z^{(j)})$ ⓘ
5end for; return $\boldsymbol{e}$ ⓘ

§5.4 → §5.4.1 Point estimation

Statistics gives the tools to formalize generalization. The atom is the point estimator — any function of the data offering a single “best” guess of a quantity:

Term by term

\hat{\theta}_m = g(\boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)})

$g$	ANY function of the data qualifies as a point estimator (statistic) — the definition demands nothing, so quality must be analyzed separately (bias §5.4.2, variance §5.4.3)	arbitrary
$\hat{\theta}_m$	the estimate — frequentist view: the true θ is FIXED but unknown; θ̂ is a RANDOM VARIABLE because the data is random	random
$\hat{f}$	function estimation (predict y from x, y = f(x) + ε) is the same thing — a point estimator in function space, as with linear regression’s w	special case

The good-estimator vocabulary — bias, variance, and their trade-off — is next.

Check yourself — regularization and validation

1.In the fig 5.5 widget (degree fixed at 9), predict the three λ regimes.

2.What is the book's exact definition of regularization?

3.Why can't capacity hyperparameters (degree, λ) be learned on the training set?

4.Why must the validation set come from the training data rather than the test set?

5.What does k-fold cross-validation buy, and at what price — plus its statistical caveat?

5 questions