Momentum · Nesterov Momentum · Deep Learning Interactive

§8.3.2 Momentum

Plain SGD crawls through the ill-conditioned canyons of §8.2.1. Momentum (Polyak, 1964) accelerates it, especially when the curvature is high, the gradients are small but consistent, or the gradients are noisy. The idea: accumulate an exponentially-decaying moving average of past gradients — a velocity — and keep moving in its direction.

Term by term

\boldsymbol{v} \leftarrow \alpha\boldsymbol{v} - \epsilon\,\boldsymbol{g}, \qquad \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \boldsymbol{v}

$\alpha\boldsymbol{v}$	keep a fraction α ∈ [0,1) of the previous velocity — an exponentially-decaying memory of past gradients (think of the reach as 1/(1−α))	the memory
$-\epsilon\boldsymbol{g}$	add the current negative gradient as a force. Plain gradient descent is just the α = 0 case (no memory)	the force
$\frac{\epsilon\lVert\boldsymbol{g}\rVert}{1-\alpha}$	if the gradient stays constant, the speed builds to this TERMINAL VELOCITY — α = 0.9 gives a 10× larger step than plain GD	top speed

The payoff is clearest on the canyon: plain GD zig-zags across the steep walls while creeping along the floor, but momentum lets the consistent floor-direction gradients accumulate while the alternating wall-gradients cancel. Switch the optimizer and raise $\alpha$ to watch it cut straight down the valley (fig 8.5):

Fig 8.5 (interactive) — on a poorly-conditioned quadratic, plain GD zig-zags but momentum accumulates velocity along the canyon floor. Try κ = 12, then switch GD → momentum.

κ12.0ε0.60

show Newton's method (uses H⁻¹ — one jump)

step 0/80 · f = 85.75

Momentum builds velocity along the canyon floor (consistent gradients accumulate) while the alternating wall-gradients cancel out — terminal speed ε‖g‖/(1−α) = 10.0× the plain-GD step.

The physics: a hockey puck with drag

Momentum simulates a particle under Newtonian dynamics. Two forces act on it. One is the negative gradient $-\nabla J$ — like a hockey puck sliding downhill on ice, gathering speed. But friction-free ice means it would oscillate across a valley forever, so we add a second force proportional to $-\boldsymbol{v}$ : viscous drag. That integer power of velocity is just right — turbulent drag ( $\propto v^2$ ) is too weak to stop the particle, dry friction ( $\propto v^0$ ) is so strong it halts before reaching the minimum, but viscous drag lets the gradient keep it moving while still damping pointless oscillation.

§8.3.3 Nesterov momentum

Nesterov momentum (Sutskever et al., 2013) is a small but clever tweak: evaluate the gradient after applying the current velocity — at the look-ahead point $\tilde{\boldsymbol{\theta}} = \boldsymbol{\theta} + \alpha\boldsymbol{v}$ — rather than at the current point. It is a correction: since momentum is about to carry you to $\theta + \alpha v$ anyway, measure the slope there.

Algorithm 8.3 — SGD with Nesterov momentum (click lines)

1Require: learning rate $\epsilon$ , momentum $\alpha$ ; initial $\boldsymbol{\theta}$ , velocity $\boldsymbol{v}$ ⓘ
2while stopping criterion not met do
3sample a minibatch $\{\boldsymbol{x}^{(1)},\dots,\boldsymbol{x}^{(m)}\}$ with targets $\boldsymbol{y}^{(i)}$
4apply interim update: $\tilde{\boldsymbol{\theta}} \leftarrow \boldsymbol{\theta} + \alpha\boldsymbol{v}$ ⓘ
5compute gradient at the interim point: $\boldsymbol{g} \leftarrow \frac{1}{m}\nabla_{\tilde{\boldsymbol{\theta}}} \sum_i L(f(\boldsymbol{x}^{(i)};\tilde{\boldsymbol{\theta}}), \boldsymbol{y}^{(i)})$ ⓘ
6velocity update: $\boldsymbol{v} \leftarrow \alpha\boldsymbol{v} - \epsilon\boldsymbol{g}$
7apply update: $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \boldsymbol{v}$

The look-ahead lets Nesterov anticipate an overshoot and correct for it — toggle to “Nesterov” in the widget above to feel the difference. In the convex batch setting this improves the convergence rate from $O(1/k)$ to $O(1/k^2)$ (Nesterov, 1983); in the stochastic case, unfortunately, it does not improve the rate.

Next: how to initialize the parameters, and adaptive per-parameter learning rates (AdaGrad, RMSProp) — initialization & adaptive learning rates.

Check yourself — momentum and Nesterov momentum

1.What does the momentum update v ← αv − εg do, and how does it help on an ill-conditioned canyon?

2.With momentum and a constant gradient g, what is the terminal (maximum) step size, and what does α = 0.9 give?

3.In the physics analogy, why is 'viscous drag' (a force proportional to −v) the right friction to add?

4.What single thing distinguishes Nesterov momentum from standard momentum?

5.How much does Nesterov momentum improve convergence, and in which setting?

5 questions