§7.1.2–7.4L¹ Regularization & Sparsity · Norm Penalties as Constraints · Under-Constrained Problems · Dataset Augmentation

§7.1.2 L¹ regularization — the road to sparsity

Swap the squared L² norm for the L¹ norm $\Omega(\boldsymbol{\theta}) = \lVert\boldsymbol{w}\rVert_1 = \sum_i |w_i|$ and the behavior changes qualitatively. The whole difference is visible in the (sub-)gradient:

Assume (as the book does) that the inputs were decorrelated so the Hessian is diagonal, $\boldsymbol{H} = \mathrm{diag}(H_{1,1},\dots,H_{n,n})$ with $H_{i,i}>0$. Then the regularized objective decouples into one independent scalar problem per weight, each with a clean closed form:

Compare the two side by side — L¹ has a dead zone that zeroes small weights, while L² only rescales:

For L², revisiting eq 7.13 under the same diagonal Hessian gives $\tilde w_i = \frac{H_{i,i}}{H_{i,i}+\alpha}w_i^*$: if $w_i^*$ was nonzero, $\tilde w_i$ stays nonzero. So L² never makes weights sparse; L¹ can. That sparsity is used for feature selection — the famous LASSO lasso Least Absolute Shrinkage and Selection Operator (Tibshirani 1995): an L¹ penalty + linear model + least-squares cost; the zeroed weights perform automatic feature selection. defined in ch. 7 — open in glossary pairs an L¹ penalty with least squares, and the weights driven to zero mark features that can be discarded.

§7.2 Norm penalties as constrained optimization

A penalty and a hard constraint are two views of the same thing. To force $\Omega(\boldsymbol{\theta}) < k$, build a generalized Lagrangian:

So weight decay confines $\boldsymbol{w}$ to a ball. Which region is set by the norm — and this is where L¹’s sparsity becomes geometric. Morph the unit ball below: L²’s smooth circle meets the objective anywhere, but L¹’s diamond has corners on the axes, and the constrained optimum tends to land on a corner — where some coordinates are exactly zero:

§7.3 Regularization for under-constrained problems

Sometimes regularization is not optional — it is what makes the problem well-defined. Many linear models (linear regression, PCA) invert $\boldsymbol{X}^\top\boldsymbol{X}$, which is singular when a direction has no variance or when there are fewer examples than features. Adding the penalty inverts $\boldsymbol{X}^\top\boldsymbol{X} + \alpha\boldsymbol{I}$ instead — always invertible. Likewise, logistic regression on linearly separable data under-constrained problem A problem where XᵀX is singular or the optimum diverges (e.g. logistic regression on linearly separable data); regularization (invert XᵀX+αI, or weight decay) makes it well-defined and convergent. defined in ch. 7 — open in glossary has no finite optimum (if $\boldsymbol{w}$ separates the classes, so does $2\boldsymbol{w}$ with higher likelihood), so SGD drives $\lVert\boldsymbol{w}\rVert\to\infty$ until overflow; weight decay halts it. This connects to a familiar object:

§7.4 Dataset augmentation

The best way to generalize better is more data — so make fake data. This is easiest for classification: a classifier must be invariant to many transforms of $\boldsymbol{x}$, so generate new $(\boldsymbol{x}, y)$ pairs by transforming the inputs. For object recognition, translating an image a few pixels, rotating, or scaling it yields large gains. The one rule: never apply a transform that changes the correct label. Toggle the transforms below — three are safe, one is a trap:

Some invariances are hard to synthesize (out-of-plane rotation is not a pixel operation), but augmentation also helps speech recognition. Crucially, injecting noise noise injection Adding random noise to inputs (≈ a weight-norm penalty), to weights (≈ flat-minima / Bayesian uncertainty), or to hidden units (dropout) as a form of regularization / augmentation. defined in ch. 7 — open in glossary into the inputs is itself a form of augmentation: neural nets are surprisingly fragile to input noise, so training with it improves robustness (this is the idea behind the denoising autoencoder, and — applied to hidden units — behind dropout). One caveat for fair benchmarking: because augmentation can dramatically cut error, comparing two algorithms is only valid when both use the same augmentation scheme.

Next: noise robustness proper, label smoothing, semi-supervised and multi-task learning — noise, semi-supervised & multi-task.