§4.3.1Beyond the Gradient: Jacobian and Hessian Matrices

For vector→vector functions, all the partial derivatives assemble into the Jacobian matrix jacobian matrix All partial derivatives of a vector→vector function: Jᵢⱼ = ∂f(x)ᵢ/∂xⱼ. defined in ch. 4 — open in glossary : for $f: \mathbb{R}^m \to \mathbb{R}^n$, $J \in \mathbb{R}^{n \times m}$ with $J_{i,j} = \partial f(\boldsymbol{x})_i / \partial x_j$. And derivatives of derivatives — second derivatives — tell us how the slope itself will change as we move: they measure curvature curvature What second derivatives measure: whether a gradient step under/over-delivers vs the linear prediction (negative: better; positive: worse, can overshoot). defined in ch. 4 — open in glossary , i.e. whether a gradient step will deliver as much improvement as the slope promised.

The Hessian

With multiple input dimensions the second derivatives collect into the Hessian matrix hessian matrix H(f)ᵢⱼ = ∂²f/∂xᵢ∂xⱼ — the Jacobian of the gradient; symmetric (a.e.), so real eigenvalues + orthogonal eigenbasis; dᵀHd is directional curvature. defined in ch. 4 — open in glossary :

How does curvature interact with a gradient step? Take the second-order Taylor approximation around $\boldsymbol{x}^{(0)}$ and substitute the step $\boldsymbol{x} = \boldsymbol{x}^{(0)} - \epsilon\boldsymbol{g}$:

The second derivative test

At a critical point ($\nabla f = 0$), the Hessian’s eigenvalues decide what you’re standing on — the multivariate second derivative test second derivative test At a critical point: H positive definite → local min; negative definite → local max; mixed eigenvalue signs → saddle; a zero eigenvalue → inconclusive. defined in ch. 4 — open in glossary :

Ill-conditioning: the canyon

In multiple dimensions there is a different second derivative in every direction; the condition number condition number How sensitive a function's output is to input perturbation; for a matrix, maxᵢⱼ|λᵢ/λⱼ| — large means error amplification, an intrinsic property. defined in ch. 4 — open in glossary of the Hessian measures how much they disagree. A poorly conditioned Hessian is a canyon: one direction curves steeply (the walls, $\lambda_{\max}$), another barely at all (the floor, $\lambda_{\min}$). Gradient descent suffers doubly — it wastes steps zigzagging down the walls, and the step size must stay small enough for the walls, making floor progress glacial. This is the book’s fig 4.6, live:

Newton’s method — using the Hessian to steer

The zigzag exists because gradient descent is blind to curvature. Newton’s method newtons method Jump to the critical point of the local second-order Taylor: x* = x⁽⁰⁾ − H⁻¹∇f; exact in one step for positive definite quadratics; attracted to saddles. defined in ch. 4 — open in glossary solves the local second-order Taylor model exactly:

What guarantees can we get?

DL objectives are too complicated for most classical guarantees. Two partial refuges: Lipschitz continuity lipschitz continuous |f(x) − f(y)| ≤ L‖x−y‖₂: rate of change bounded by L — the weak guarantee most DL objectives can be given. defined in ch. 4 — open in glossary — $|f(\boldsymbol{x}) - f(\boldsymbol{y})| \le L\|\boldsymbol{x} - \boldsymbol{y}\|_2$, a weak bound (“small input change ⟹ small output change”) that many DL problems can be given with minor modifications — and convex optimization convex optimization Optimization restricted to functions with positive semidefinite Hessian everywhere: no saddles, every local minimum global; in DL mostly a subroutine/analysis tool. defined in ch. 4 — open in glossary , the most successful specialized field: Hessian positive semidefinite everywhere, hence no saddles and every local minimum global. Most DL problems can’t be expressed convexly, so convexity survives mainly as a subroutine and a source of analysis ideas.