§7.5–7.7Noise Robustness · Label Smoothing · Semi-Supervised & Multi-Task Learning

§7.5 Noise robustness

Section 7.4 injected noise into the inputs as augmentation; for infinitesimal variance that turns out to be equivalent to a penalty on the weight norm. But noise injection noise robustness Training with injected noise so the learned function is insensitive to small perturbations. Input noise ≈ a weight-norm penalty; weight noise adds an η·E‖∇_W ŷ‖² term that pushes toward FLAT minima (robust to weight perturbation). defined in ch. 7 — open in glossary can be much more powerful, especially when the noise goes on the weights — a common trick in recurrent nets, interpretable as a stochastic Bayesian treatment of weight uncertainty. Take a regression net trained on squared error, and add a small perturbation $\boldsymbol{W}\sim\mathcal{N}(0,\eta\boldsymbol{I})$ to the weights each step. For small $\eta$, minimizing the noisy cost is equivalent to minimizing the clean cost plus a regularizer:

Explore what “prefers flat minima” means: both wells below are equally deep, but a weight perturbation of $\pm\sigma$ raises the loss far more in the sharp one:

§7.5.1 Smoothing the labels

Datasets contain label mistakes, so blindly maximizing $\log p(y\mid \boldsymbol{x})$ on a wrong $y$ is harmful. Label smoothing label smoothing Replace hard 0/1 softmax targets with soft ε/(k−1) and 1−ε, so maximum likelihood need not chase unreachable extreme probabilities (which would grow weights forever); accounts for label mistakes. defined in ch. 7 — open in glossary softens the classification targets: instead of a hard one-hot, use $1-\epsilon$ for the correct class and $\frac{\epsilon}{k-1}$ for each of the other $k-1$ classes, then apply the usual cross-entropy. Slide $\epsilon$:

The motivation is subtle: a softmax can never output exactly 0 or 1, so maximum-likelihood training with hard targets never converges — it just keeps driving the weights larger, making ever more extreme predictions. Label smoothing gives the model a reachable target, discouraging that runaway without discouraging correct classification. It has been used since the 1980s and features in modern nets (Szegedy et al., 2015).

§7.6 Semi-supervised learning

Semi-supervised learning semi-supervised learning Use both unlabeled examples from P(x) and labeled examples from P(x,y) to estimate P(y|x) — e.g. share parameters between a generative model of P(x) and a discriminative model of P(y|x), so unlabeled data shapes the representation. defined in ch. 7 — open in glossary uses both unlabeled examples from $P(\boldsymbol{x})$ and labeled examples from $P(\boldsymbol{x}, y)$ to estimate $P(y\mid\boldsymbol{x})$. In deep learning this usually means learning a representation $\boldsymbol{h}=f(\boldsymbol{x})$ where examples of the same class land close together — unsupervised structure (tight input-space clusters) becomes a cue for grouping. Rather than bolting separate unsupervised and supervised components together, one can build a model whose generative part (for $P(\boldsymbol{x})$ or $P(\boldsymbol{x}, y)$) shares parameters with the discriminative part (for $P(y\mid\boldsymbol{x})$), then trade off the criteria $-\log P(y\mid\boldsymbol{x})$ and $-\log P(\boldsymbol{x})$. The generative term acts as a prior: it asserts that the structure of $P(\boldsymbol{x})$ is tied to the structure of $P(y\mid\boldsymbol{x})$, and tuning how much of it to include can beat purely generative or purely discriminative training.

§7.7 Multi-task learning

Multi-task learning multi-task learning Improve generalization by sharing lower-layer parameters across several tasks (a common factor pool) while keeping task-specific upper layers; pooled data constrains the shared part toward values that generalize. defined in ch. 7 — open in glossary improves generalization by pooling the examples of several tasks — sharing part of the model acts as a soft constraint that pushes the shared parameters toward values that generalize. The classic architecture (fig 7.2) sends a common input through shared lower layers to a representation $\boldsymbol{h}^{(\text{shared})}$, then splits into task-specific heads:

The parameters divide into two kinds: task-specific ones (the upper layers, which benefit only from their own task’s examples) and generic shared ones (the lower layers, which benefit from the pooled data of all tasks). The shared part gets far more effective examples, so its statistical strength — and generalization — improves, provided the tasks really do share something. The underlying prior belief: among the factors that explain the variation across the tasks’ data, some are shared.

Next: early stopping (the most common regularizer of all) and parameter tying & sharing — early stopping & parameter sharing.