§4.1–4.3Overflow & Underflow · Poor Conditioning · Gradient-Based Optimization

Machine learning runs on numerical computation: iterative methods that update estimates rather than deriving symbolic solutions. The fundamental difficulty: infinitely many reals must live in finitely many bit patterns, so almost every number carries rounding error — and error compounds across operations, making theoretically correct algorithms fail in practice.

§4.1 Overflow and underflow

Underflow underflow Numbers near zero rounding to exactly zero — then division and log blow up (÷0, log 0 = −∞). defined in ch. 4 — open in glossary : numbers near zero round to exactly zero — and many functions change qualitatively at zero (division raises or returns NaN; log 0 = −∞, which becomes NaN downstream). Overflow overflow Large-magnitude numbers approximated as ±∞; further arithmetic turns them into NaN. defined in ch. 4 — open in glossary : large magnitudes become ±∞, and further arithmetic turns them into NaN.

The canonical function needing stabilization is the softmax softmax softmax(x)ᵢ = exp(xᵢ)/Σⱼexp(xⱼ): turns a real vector into multinoulli probabilities; stabilized by subtracting maxᵢxᵢ (shift-invariant). defined in ch. 4 — open in glossary , used to produce multinoulli probabilities:

The fix exploits shift-invariance — adding a scalar to every input does not change the value: evaluate $\mathrm{softmax}(\boldsymbol{z})$ with $\boldsymbol{z} = \boldsymbol{x} - \max_i x_i$. The largest argument to exp becomes 0 (no overflow) and the denominator contains a 1 (no divide-by-zero). Try to break both versions:

One residue remains: the numerator can still underflow, so naively taking $\log(\mathrm{softmax}(x))$ can produce $-\infty$ — log softmax needs its own stabilized routine (same trick). In practice, most readers lean on stable low-level libraries; Theano-style packages even detect and auto-stabilize common unstable expressions.

§4.2 Poor conditioning

Conditioning measures how rapidly a function’s output changes under small input perturbations — fast change means input rounding error becomes large output error. For $f(\boldsymbol{x}) = \boldsymbol{A}^{-1}\boldsymbol{x}$:

§4.3 Gradient-based optimization

Optimization: minimize (or maximize — just minimize $-f$) an objective function objective function The function being minimized/maximized (= criterion; when minimized also called cost, loss, or error function); optimum written x* = arg min f(x). defined in ch. 4 — open in glossary , a.k.a. criterion — or, when minimized, cost / loss / error function (this book uses the terms interchangeably). The optimum gets a star: $x^* = \arg\min f(x)$.

The derivative $f'(x)$ is the slope: $f(x + \epsilon) \approx f(x) + \epsilon f'(x)$ — it tells us how to change $x$ to improve $y$: $f(x - \epsilon\,\mathrm{sign}(f'(x))) < f(x)$ for small $\epsilon$. Stepping opposite the derivative is gradient descent gradient descent (Steepest descent, Cauchy 1847) iterate x′ = x − ε∇ₓf(x): small steps against the gradient until it (nearly) vanishes. defined in ch. 4 — open in glossary (Cauchy, 1847):

Where $f'(x) = 0$, the derivative says nothing — critical (stationary) points critical point (Stationary point) where the derivative/gradient is zero: local minimum, local maximum, or saddle point. defined in ch. 4 — open in glossary , of three kinds:

Deep learning objectives have many suboptimal local minima and saddle points in flat regions, so we settle for finding $f$ very low — not provably minimal (fig 4.3’s landscape: a near-optimal local minimum is an acceptable halting point; a poor one should be avoided; the global one may be unreachable).

The multivariate case: which direction is downhill?

With inputs $\boldsymbol{x} \in \mathbb{R}^n$, partial derivatives $\partial f/\partial x_i$ assemble into the gradient gradient ∇ₓf(x): the vector of all partial derivatives; points directly uphill, so −∇f points directly downhill. defined in ch. 4 — open in glossary $\nabla_{\boldsymbol{x}} f$, and the slope along a unit vector $\boldsymbol{u}$ is the directional derivative $\boldsymbol{u}^\top \nabla_{\boldsymbol{x}} f$. The best direction falls out in three moves: