Architecture Extras · Back-Propagation · Computational Graphs · The Chain Rule

§6.4.2 Beyond the plain chain

Real networks show more diversity than a simple chain. Two common departures:

Skip connections run from layer $i$ to layer $i + 2$ or beyond, making it easier for the gradient to flow from the output back toward the input (the seed of residual networks).
Sparse connections wire each unit to only a small subset of the next layer’s units — fewer parameters and less computation, but problem-dependent. Convolutional networks (chapter 9) use a highly effective sparse, shared pattern for vision.

Specific architectural strategies for each domain fill later chapters; this chapter’s remaining job is the algorithm that makes any of them trainable.

§6.5 Back-propagation

When a network maps input $\boldsymbol{x}$ to output $\hat{\boldsymbol{y}}$ (and on to a scalar cost $J$ ), information flows forward — forward propagation . The back-propagation algorithm (Rumelhart, 1986) then lets information from the cost flow backward to compute the gradient.

Three things back-propagation is NOT

(1) It is not the whole learning algorithm — it only computes the gradient; SGD (or another optimizer) does the learning with it. (2) It is not specific to neural networks — it can differentiate essentially any function. (3) It is not just “the chain rule” — the chain rule gives the formula; back-propagation is a specific, highly efficient ORDER of applying it (§6.5.3). Computing the gradient analytically is easy; evaluating it cheaply is the contribution.

§6.5.1 Computational graphs

To describe back-propagation precisely we need a computational graph : each node is a variable (scalar, vector, matrix, or tensor), and each operation is a simple function of one or more variables, producing one output. A directed edge $x \to y$ means $y$ is computed by an operation applied to $x$ . Here is the graph for logistic regression, $\hat{y} = \sigma(\boldsymbol{x}^\top\boldsymbol{w} + b)$ (fig 6.8b) — intermediate values that have no name in the algebra get named $u^{(i)}$ :

Fig 6.8b — the computational graph for ŷ = σ(xᵀw + b); click a node

Each operation node produces one intermediate value; edges show data dependencies.

§6.5.2 The chain rule of calculus

Back-propagation computes the chain rule of calculus — not the chain rule of probability from §3.6. For $y = g(x)$ and $z = f(y)$ , the scalar rule generalizes to vectors and tensors:

Term by term

\frac{dz}{dx} = \frac{dz}{dy}\frac{dy}{dx} \;\;\longrightarrow\;\; \nabla_{\boldsymbol{x}} z = \left(\frac{\partial \boldsymbol{y}}{\partial \boldsymbol{x}}\right)^{\!\top} \nabla_{\boldsymbol{y}} z

$\partial\boldsymbol{y}/\partial\boldsymbol{x}$	the Jacobian matrix of g (n×m for g: ℝᵐ → ℝⁿ) — from §4.3.1	Jacobian
$\nabla_{\boldsymbol{y}} z$	the gradient already computed at the downstream variable y	gradient
$(\cdot)^\top \nabla_{\boldsymbol{y}} z$	the gradient at x = a JACOBIAN-GRADIENT PRODUCT. Back-propagation performs exactly one such product per operation in the graph	the core step

What this really says

A gradient is carried backward through one operation by left-multiplying by that operation’s Jacobian (transposed). Tensors change nothing conceptually — flatten to a vector, apply the same product, reshape back. So the whole of back-propagation is: start with $\nabla_{\hat y} J$ , and repeatedly multiply by Jacobians going backward. The only cleverness is the order — which the next section makes precise, and which is the difference between linear-time and exponential-time.

Check yourself — graphs and the chain rule

1.What does a skip connection (layer i → i+2) buy?

2.Which statement about back-propagation is correct?

3.In a computational graph, what are nodes and operations?

4.The vector chain rule is ∇_x z = (∂y/∂x)ᵀ ∇_y z. What single operation does back-propagation perform per graph operation?

5.The chain rule of CALCULUS should not be confused with which similarly-named rule?

5 questions