Welcome to Part II — the established, essentially-solved technology of modern deep networks. It opens with the quintessential model: deep feedforward networks feedforward network (MLP) the quintessential deep model: y = f(x;θ) as a composed chain f³∘f²∘f¹ with NO feedback; approximates a target f*. defined in ch. 6 — open in glossary , also called feedforward neural networks or multilayer perceptrons (MLPs) multilayer perceptron A function composed of many simpler functions; the quintessential deep model (feedforward deep network). defined in ch. 1 — open in glossary . The goal is to approximate some function — for a classifier, maps an input to a category — by learning the parameters of a mapping that best approximates it.
Three words, three ideas:
- Feedforward — information flows one way: from , through intermediate computations, to , with no feedback connections. (Add feedback and you get recurrent networks, chapter 10.)
- Network — the model is a chain of composed functions , described by a directed acyclic graph.
- Neural — loosely inspired by neuroscience: each layer is vector-valued, and each element plays the role of a neuron receiving input from many others. But the honest framing is function approximation machines designed for statistical generalization, only occasionally borrowing from the brain.
The chain f³∘f²∘f¹. The output layer output layer The final layer of a feedforward net, driven directly by the training targets. defined in ch. 6 — open in glossary is driven directly by the labels; the depth depth The length of the composed-function chain — the source of the name 'deep learning'. (First introduced ch01.) defined in ch. 6 — open in glossary of the chain is the source of the name “deep learning”; each layer’s width width The dimensionality of a hidden layer — how many parallel unit/neuron activations it holds. defined in ch. 6 — open in glossary is how many parallel units it holds. The training data says what the output must do — but never what each hidden layer should compute.
Why we need more than a linear model
Linear models (linear/logistic regression) fit efficiently and reliably — in closed form or by convex optimization — but their capacity is confined to linear functions, so they cannot capture the interaction between two input variables. The remedy: apply the linear model not to but to a nonlinear transform — a set of features, a new representation of . The whole question becomes how to choose φ:
| How φ is obtained | Trade-off | |
|---|---|---|
| 1. Generic φ | a fixed, generic map (RBF kernel) | fits anything, generalizes poorly |
| 2. Hand-engineer φ | human-designed features | powerful but decades of per-task labor |
| 3. Learn φ (deep learning) | θ learned from a broad family φ(x; θ) | gives up convexity, wins in practice |
The recurring theme of the whole book
“Improve the model by learning the features” is not confined to feedforward networks — it is the thread running through every model in this book. Feedforward networks apply it to learning deterministic mappings without feedback; later chapters apply the same principle to stochastic mappings, feedback (recurrence), and full probability distributions. The hidden layer is where a network stops consuming features and starts inventing them.
The chapter ahead
Deploying a feedforward network reuses every linear-model design decision — the optimizer, the cost function, the form of the output units (§6.2) — plus decisions unique to having hidden layers: the activation functions for hidden units (§6.3), the architecture (how many layers, how wide, how connected — §6.4), and the algorithm that makes learning tractable at all: back-propagation for computing gradients of these composed functions (§6.5). We begin, as the book does, with the smallest network that does something a linear model cannot: learning XOR.
Check yourself — what a feedforward network is
1.Why are these networks called 'feedforward', and what changes that name?
2.In f = f³(f²(f¹(x))), which layers' behavior does the training data directly specify?
3.What can't a linear model do that motivates feature maps φ(x)?
4.Deep learning's strategy is to LEARN φ. What does it give up, and what does it gain over the other two strategies?
5.Which best describes the book's stance on the 'neural' in neural networks?