Differentiation Beyond Deep Learning · Higher-Order Derivatives · History

§6.5.9 Back-propagation is one flavour of automatic differentiation

The deep-learning community reinvented some wheels. The broader field is automatic differentiation, and back-propagation is one special case: reverse mode accumulation . The alternative, forward mode accumulation , evaluates the chain rule’s factors in the opposite order. Which is cheaper depends entirely on shape — think of the Jacobians as a matrix chain $\boldsymbol{A}\boldsymbol{B}\boldsymbol{C}\boldsymbol{D}$ :

Term by term

\boldsymbol{A}\boldsymbol{B}\boldsymbol{C}\boldsymbol{D} \quad \text{— multiply right-to-left (reverse) or left-to-right (forward)?}

$\text{reverse (backprop)}$	multiply from the RIGHT: if D is a column vector (few outputs, MANY inputs — like ∇ of a scalar loss over millions of params), every product is a cheap matrix-VECTOR product	few out, many in
$\text{forward mode}$	multiply from the LEFT: cheaper when A has FEWER rows than D has columns — i.e. #outputs > #inputs. Used e.g. for real-time RNN gradients, trading memory for compute	many out, few in
$\text{NP-complete}$	finding the globally cheapest evaluation order is NP-complete; libraries (Theano/TensorFlow) use pattern-matching heuristics to simplify	the hard problem

Aside — backprop won’t simplify your algebra

Back-propagation is mechanical, not clever. For a softmax cross-entropy loss, a human sees the gradient collapses to $q_i - p_i$ ; pure back-propagation instead grinds through every exp/log/division node in the graph. Libraries like Theano add algebraic-simplification passes on top precisely to recover such shortcuts. The deep-learning approach — explicit graph data structures with per-operation bprop — trades the generality of differentiating raw source code for the ability to hand-tune each operation’s backward rule for speed and stability.

§6.5.10 Higher-order derivatives — never form the Hessian

In deep learning we rarely want a single second derivative; we want properties of the Hessian. But with $n$ parameters (often billions), the $n \times n$ Hessian is infeasible even to store. The workaround is Krylov methods — iterative techniques (approximate inversion, eigenvectors) that need only Hessian-vector products, never the matrix itself:

Term by term

\boldsymbol{H}\boldsymbol{v} = \nabla_{\boldsymbol{x}}\big[ (\nabla_{\boldsymbol{x}} f(\boldsymbol{x}))^\top \boldsymbol{v} \big]

$(\nabla_{\boldsymbol{x}} f)^\top\boldsymbol{v}$	the inner gradient dotted with v — a scalar. Both gradients here are ordinary back-propagation calls	inner
$\nabla_{\boldsymbol{x}}[\cdots]$	differentiate that scalar again → the Hessian-vector product Hv, WITHOUT ever building H	outer
$\boldsymbol{H}\boldsymbol{e}^{(i)}$	if you truly need a column of H, use v = the i-th one-hot vector — but that is rarely worth it	a column, if forced

§6.6 Historical notes

Feedforward networks are the culmination of centuries of work on function approximation. The chain rule dates to the 17th century (Leibniz, 1676); gradient descent to the 19th (Cauchy, 1847); efficient chain-rule application via dynamic programming appeared in 1960s–70s control theory, and Werbos (1981) proposed it for neural nets. Back-propagation was rediscovered independently (LeCun, Parker, Rumelhart, 1986); the Parallel Distributed Processing book popularized it and launched connectionism, with distributed representation as a central idea.

What actually changed since the 1980s

The core is unchanged — same back-propagation, same gradient descent. Most 1986→2015 gains came from bigger datasets (less of a generalization burden) and bigger networks (faster hardware, better software). Only two algorithmic changes mattered much:
1. MSE → cross-entropy loss — cured the slow learning of saturated sigmoid/softmax outputs.
2. sigmoid → ReLU hidden units — piecewise-linear units learn far more easily than curved, two-sided-saturating ones. Jarrett et al. (2009): a rectifying nonlinearity was “the single most important factor” in their recognition system.

A biological footnote closes the chapter: ReLU was itself neuroscience-motivated — biological neurons are inactive for many inputs, roughly linear when active, and mostly inactive (sparse) — so neuroscience still occasionally guides deep learning. And the field’s self-image flipped: feedforward nets, distrusted from 2006–2012 and thought to need propping up by probabilistic models, are now the powerful default — used to build probabilistic models (VAEs, GANs in chapter 20). In 2006 unsupervised learning supported supervised learning; today, ironically, supervised learning supports unsupervised.

That completes the neural-network family. Part II now turns to how to use them — starting with how to keep them from overfitting: chapter 7, Regularization.

Check yourself — automatic differentiation, higher-order derivatives, history

1.How does back-propagation relate to the broader field of automatic differentiation?

2.Picture the Jacobians as a matrix chain ABCD. When is reverse mode (back-propagation) the cheaper multiplication order?

3.Is finding the globally cheapest order to evaluate the chain-rule products easy?

4.With billions of parameters, why do we compute Hessian-vector products Hv = ∇ₓ[(∇ₓ f)ᵀv] instead of the Hessian H itself?

5.The book argues that most 1986→2015 gains came from bigger data and bigger models, with only two algorithmic changes mattering much. What were they?

5 questions