Adam · Choosing an Optimizer · Newton's Method

§8.5.3 Adam

Adam (“adaptive moments”) combines the two ideas we just met: momentum (a first-moment estimate of the gradient) and RMSProp (a second-moment estimate of the squared gradient) — plus a bias correction that RMSProp lacks:

Algorithm 8.7 — Adam (click lines)

1Require: step $\epsilon$ (0.001); decay rates $\rho_1{=}0.9,\ \rho_2{=}0.999$ ; $\delta{=}10^{-8}$ ; initial $\boldsymbol{\theta}$ ⓘ
2initialize moments $\boldsymbol{s}\leftarrow 0,\ \boldsymbol{r}\leftarrow 0$ ; time $t\leftarrow 0$ ⓘ
3while stopping criterion not met do
4compute gradient $\boldsymbol{g}$ ; $\ t \leftarrow t+1$
5first moment: $\boldsymbol{s} \leftarrow \rho_1\boldsymbol{s} + (1-\rho_1)\boldsymbol{g}$ ⓘ
6second moment: $\boldsymbol{r} \leftarrow \rho_2\boldsymbol{r} + (1-\rho_2)\boldsymbol{g}\odot\boldsymbol{g}$ ⓘ
7bias-correct: $\hat{\boldsymbol{s}} \leftarrow \dfrac{\boldsymbol{s}}{1-\rho_1^t},\quad \hat{\boldsymbol{r}} \leftarrow \dfrac{\boldsymbol{r}}{1-\rho_2^t}$ ⓘ
8update: $\Delta\boldsymbol{\theta} = -\epsilon\,\dfrac{\hat{\boldsymbol{s}}}{\sqrt{\hat{\boldsymbol{r}}}+\delta}$ ; $\ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \Delta\boldsymbol{\theta}$ ⓘ

§8.5.4 Which optimizer?

There is no consensus. The adaptive-rate family is robust, but no single method wins across tasks — the choice often comes down to which one you know well enough to tune. Here is the lineage:

The optimizer family — what each one adapts
	what it adds	notes
SGD	nothing — step −ε·g with a decaying global ε	the baseline; robust and cheap
SGD + momentum	a velocity (a 1st-moment memory of gradients)	accelerates down canyons; adds α
AdaGrad	per-parameter rate ∝ 1/√(Σ g²)	convex-friendly, but the full-history sum shrinks the rate too early for deep nets
RMSProp	per-parameter rate from an EMA of g²	fixes AdaGrad's over-decay; a go-to method
Adam	1st + 2nd moments (momentum + RMSProp) + bias correction	robust defaults; a common first choice

Dotted-underlined cells have explanations — click one.

§8.6.1 Newton’s method

All those methods use only the gradient. Second-order methods use the curvature too. Newton’s method fits a quadratic (a second-order Taylor expansion) at the current point and jumps straight to its minimum:

Term by term

\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 - \boldsymbol{H}^{-1}\,\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}_0)

$\boldsymbol{H}^{-1}\nabla J$	rescale the gradient by the INVERSE Hessian — for a positive-definite quadratic this lands exactly on the minimum in a single step (no learning rate to tune)	the jump
$[\boldsymbol{H} + \alpha\boldsymbol{I}]^{-1}$	danger: if H has NEGATIVE eigenvalues (near a saddle), the step points the WRONG way. Regularize by adding α to the diagonal (Levenberg–Marquardt)	saddle fix
$O(k^3)$	the k×k Hessian must be formed AND inverted every iteration — with k in the millions this is infeasible, so we approximate it (CG, BFGS)	the cost

You have already seen Newton’s one-jump behavior: in the optimizer widget, tick “show Newton’s method” — while gradient descent zig-zags down the canyon, Newton uses $\boldsymbol{H}^{-1}$ to leap straight to the minimum:

Newton vs gradient descent on an ill-conditioned bowl — toggle "show Newton's method" to see the single H⁻¹ jump straight to the minimum

κ10.0ε0.50

show Newton's method (uses H⁻¹ — one jump)

step 0/80 · f = 93.10

Canyon regime: GD zigzags down the steep walls (λmax direction) while crawling along the floor (λmin direction) — ε must stay small for the walls, so progress along the floor is slow. Try the momentum optimizer.

Why Newton’s method does not scale to deep nets

Two problems. First, Newton’s method is only safe when the Hessian is positive definite — but deep loss surfaces are riddled with saddle points (§8.2.3), where negative eigenvalues make the update move uphill. The regularized $[\boldsymbol{H}+\alpha\boldsymbol{I}]^{-1}$ helps only if the negative eigenvalues are small; strong negative curvature forces $\alpha$ so large that Newton degrades to plain gradient descent with step $1/\alpha$ . Second, the cost: the Hessian has $k^2$ entries and costs $O(k^3)$ to invert every iteration — with $k$ in the millions, hopeless. The next section’s methods (conjugate gradients, BFGS) chase Newton’s benefits while sidestepping the inverse Hessian.

Next: conjugate gradients, BFGS, and the meta-algorithms — conjugate gradients, BFGS & meta-algorithms.

Check yourself — Adam, optimizer choice, and Newton's method

1.What does Adam combine, and what does its bias correction fix?

2.Which statement about choosing an optimizer is accurate?

3.What does Newton's update θ* = θ₀ − H⁻¹∇J do on a positive-definite quadratic?

4.Why is Newton's method dangerous near saddle points, and what is the standard fix?

5.Why does Newton's method not scale to large neural networks, even setting saddles aside?

5 questions