§6.1–6.2Example: Learning XOR · Gradient-Based Learning

§6.1 The smallest network that beats a linear model

To make feedforward networks concrete, the book takes the simplest task a linear model cannot do: XOR. The exclusive-or returns 1 when exactly one of $x_1, x_2$ is 1, else 0 — the target $y = f^*(\boldsymbol{x})$ over the four points $\{[0,0], [0,1], [1,0], [1,1]\}$. We train on all four (no generalization here — the only challenge is fitting), with an MSE loss (chosen purely to keep the algebra simple; MSE is a poor choice for binary data in practice — see §6.2.1).

Try the linear model $f(\boldsymbol{x}; \boldsymbol{w}, b) = \boldsymbol{x}^\top\boldsymbol{w} + b$ — solving the normal equations gives $\boldsymbol{w} = \boldsymbol{0}$, $b = \tfrac12$, i.e. 0.5 everywhere:

The fix: learn a new feature space

Add one hidden layer with two units. The network chains two functions: $\boldsymbol{h} = f^{(1)}(\boldsymbol{x}; \boldsymbol{W}, \boldsymbol{c})$ then $y = f^{(2)}(\boldsymbol{h}; \boldsymbol{w}, b)$ — the output layer is still linear regression, but applied to $\boldsymbol{h}$ instead of $\boldsymbol{x}$. $f^{(1)}$ must be nonlinear: if it were linear, the whole composition would collapse to a linear function ($\boldsymbol{w}^\top\boldsymbol{W}^\top\boldsymbol{x} = \boldsymbol{x}^\top\boldsymbol{w}'$). So we use an affine map followed by a fixed activation function activation function The fixed element-wise nonlinearity g applied after an affine map: h = g(Wᵀx + b); its choice mostly distinguishes hidden-unit types. defined in ch. 6 — open in glossary — the default rectified linear unit (ReLU) rectified linear unit g(z) = max{0, z}: the default hidden unit — near-linear, large consistent gradients, easy to optimize; dead where its activation is 0. (First introduced ch01.) defined in ch. 6 — open in glossary , $g(z) = \max\{0, z\}$:

The complete network is one clean expression:

Watch the feature space do the work

The book supplies an exact solution: $\boldsymbol{W} = \begin{bmatrix}1&1\\1&1\end{bmatrix}$, $\boldsymbol{c} = [0, -1]^\top$, $\boldsymbol{w} = [1, -2]^\top$, $b = 0$. Step through the four inputs and see why it works — the ReLU layer folds the two “1” points onto the same point in $\boldsymbol{h}$-space, where a single line finally separates the classes (fig 6.1 right):

In a real problem there might be billions of parameters and examples — you cannot guess the solution. But this one sits at a global minimum of the loss, so gradient descent could converge to it (or to one of the other equivalent solutions, depending on initialization — and it would rarely be this clean and integer-valued).

§6.2 Gradient-based learning

Training a neural network is mostly like training any other model with gradient descent: pick an optimizer, a cost function, and a model family (§5.10). The one big difference:

Everything else — computing the gradient efficiently and exactly — is what back-propagation handles. The two design choices that remain, the cost function and the output units, get special attention next, because for neural networks their interaction with saturation makes or breaks learning: cost functions and output units.