§8.2.2 Local minima — mostly a false alarm
In a convex problem any local minimum is global. Neural nets are non-convex and have enormous numbers of local minima — but most are harmless. The reason is non-identifiability model identifiability Whether enough data can pin down a single parameter setting. Neural nets are NON-identifiable, so their cost surfaces have many equivalent local minima (all equal in cost — not a real optimization problem). defined in ch. 8 — open in glossary : many different parameter settings compute the same function. The clearest case is weight-space symmetry weight-space symmetry A source of non-identifiability: permuting the m×n hidden units (n!^m ways) or, for ReLU/maxout, scaling a unit's incoming weights by α and outgoing by 1/α gives an equivalent network — so equivalent minima abound. defined in ch. 8 — open in glossary — swap two hidden units (both their incoming and outgoing weights) and the network is unchanged:
With layers of units there are such arrangements (plus ReLU/ maxout weight-rescaling symmetries), so a single function corresponds to countless equivalent minima. But they all have the same cost, so they pose no real difficulty.
A negative test for local minima
High-cost local minima could trap gradient descent — but for large modern nets, experts now believe most local minima have low cost, and it is not important to find the true global minimum. A cheap diagnostic: plot the gradient norm over time. If it does not shrink to insignificance, the problem is not local minima (nor any critical point) — many other structures also stall learning.
§8.2.3 Saddle points dominate high dimensions
For high-dimensional non-convex functions, local minima are actually rare compared to saddle points saddle point A critical point that is a minimum along some cross-sections and a maximum along others (Hessian has both signs of eigenvalues). defined in ch. 4 — open in glossary — critical points where the Hessian has both positive and negative eigenvalues (cost rises along some directions, falls along others). Here is why they proliferate: think of each eigenvalue’s sign as a coin flip; a local minimum needs all of them positive. Slide the dimension and watch that probability collapse:
A local minimum needs every eigenvalue positive — all n coins landing heads. In high dimensions that is exponentially unlikely, so saddle points vastly outnumber minima. First-order gradient descent tends to escape them; Newton's method is attracted to them.
An amazing empirical fact sharpens this: the Hessian’s eigenvalues tend to be more positive at lower cost, so low-cost critical points are likely minima while high-cost ones are likely saddles. The implications differ by method: first-order gradient descent has a small gradient near a saddle but empirically escapes it (continuous-time GD is even repelled from saddles); Newton’s method, which solves for a zero gradient, is attracted to them — which is why second-order methods have not replaced GD, and motivates the saddle-free Newton saddle-free newton A modified second-order method (Dauphin et al.) that avoids Newton's attraction to the saddle points that proliferate in high-dimensional non-convex loss surfaces. defined in ch. 8 — open in glossary method.
§8.2.4 Cliffs and exploding gradients
Deep and recurrent nets multiply many large weights together, producing cliffs — near-vertical walls in the loss surface. A normal gradient step on the face of a cliff catapults the parameters far away, destroying progress:
The fix is gradient clipping gradient clipping Thresholding the gradient magnitude before a step so a cliff cannot catapult the parameters — recalling that the gradient gives a direction, not a safe step size (§10.11.1). defined in ch. 8 — open in glossary : the gradient gives only a direction, valid in an infinitesimal region — so cap the step size when the proposed step is huge (§10.11.1).
§8.2.5 Long-term dependencies — vanishing and exploding gradients
The deepest cause of trouble in very deep (and recurrent) graphs is repeated multiplication by the same matrix. After steps that is , and its eigendecomposition tells the whole story:
Term by term
| only the eigenvalues change with depth/time — each is raised to the t-th power | the driver | |
| eigenvalues above 1 blow up (exploding gradients, the cliffs of §8.2.4 → unstable learning) | explode | |
| eigenvalues below 1 decay to 0 (vanishing gradients — you cannot tell which way to move). Only |λ|≈1 survives | vanish |
Recurrent nets reuse the same every step, so they suffer badly (deferred to §10.7); feedforward nets use a different matrix per layer and can mostly avoid it.
§8.2.6 Inexact gradients
Finally, most algorithms assume an exact gradient — but we almost always have a noisy or biased estimate inexact gradients In practice we have only noisy or biased gradient/Hessian estimates — from minibatch sampling, or from approximating intractable objectives (e.g. contrastive divergence for Boltzmann machines). defined in ch. 8 — open in glossary : minibatch sampling introduces noise, and truly intractable objectives (like the log-likelihood of a Boltzmann machine) must be approximated (e.g. contrastive divergence). Robust algorithms and easy-to-approximate surrogate losses both help.
Next: why even a perfectly-conditioned local step can fail, and the basic algorithm itself — SGD — local vs global structure & SGD.
Check yourself — local minima, saddles, cliffs, and vanishing gradients
1.Neural nets have enormous numbers of local minima from weight-space symmetry. Why are these not a real optimization problem?
2.In SaddleLab, why does P(local min) = 2⁻ⁿ collapse as the dimension n grows, and what does that imply?
3.How do first-order gradient descent and Newton's method differ in their behavior near saddle points?
4.What causes a 'cliff' in a deep or recurrent network's loss surface, and how does gradient clipping help?
5.Repeated multiplication by W gives W^t = V diag(λ)^t V⁻¹. Why does this cause vanishing and exploding gradients, and why do RNNs suffer more than feedforward nets?