§6.3.1–6.4.1Hidden Units · Architecture · Universal Approximation & Depth

§6.3.1 Rectified linear units and their generalizations

The ReLU rectified linear unit g(z) = max{0, z}: the default hidden unit — near-linear, large consistent gradients, easy to optimize; dead where its activation is 0. (First introduced ch01.) defined in ch. 6 — open in glossary $g(z) = \max\{0, z\}$ is easy to optimize precisely because it is so nearly linear: its derivative is 1 wherever the unit is active and its second derivative is 0 almost everywhere, so gradients stay large and consistent — far more useful than activations that introduce curvature. (Tip: initialize biases to a small positive value like 0.1 so units start active.) The one drawback: a unit whose activation is 0 gets no gradient. Its generalizations fix that with a nonzero slope $\alpha_i$ for $z < 0$, $h_i = \max(0, z_i) + \alpha_i \min(0, z_i)$:

• Absolute value rectification absolute value rectification A ReLU generalization with slope α = −1 for z < 0, giving g(z) = |z|; used for polarity-invariant image features. defined in ch. 6 — open in glossary fixes $\alpha = -1$ → $g(z) = |z|$ (polarity-invariant image features).
• Leaky ReLU leaky relu ReLU with a small fixed negative slope (α ≈ 0.01) so units keep receiving gradient when inactive. defined in ch. 6 — open in glossary fixes $\alpha$ small (≈ 0.01).
• Parametric ReLU (PReLU) parametric relu (PReLU) leaky-ReLU whose negative slope α is a learned parameter. defined in ch. 6 — open in glossary makes $\alpha$ a learned parameter.

Compare them all — toggle curves, drag the probe, and watch which units go “flat” (saturate) where:

Maxout

Maxout units maxout unit Outputs the max over a group of k linear pieces — learns a piecewise-linear convex activation itself; needs more regularization; resists catastrophic forgetting. defined in ch. 6 — open in glossary go further: divide $\boldsymbol{z}$ into groups of $k$ and output each group’s maximum, $g(\boldsymbol{z})_i = \max_{j \in G^{(i)}} z_j$ (toggle it on above). A maxout unit learns the activation function itself — a piecewise-linear convex function of up to $k$ pieces — so with $k = 2$ it can become ReLU, absolute-value rectification, or leaky/parametric ReLU, or something new.

§6.3.2 Sigmoid and tanh — usually the wrong hidden unit

$g(z) = \sigma(z)$ and $g(z) = \tanh(z)$ (related by $\tanh(z) = 2\sigma(2z) - 1$) saturate across most of their domain — flat, gradient-killing everywhere except near 0 — so their use as hidden units is now discouraged (as output units, paired with a log-based cost that undoes the saturation, they’re fine). When a sigmoidal unit is required, the hyperbolic tangent hyperbolic tangent tanh(z) = 2σ(2z) − 1: a sigmoidal activation preferred over the logistic sigmoid as a hidden unit because tanh(0) = 0 resembles the identity near 0. defined in ch. 6 — open in glossary beats the logistic sigmoid: $\tanh(0) = 0$ (vs $\sigma(0) = \tfrac12$), so it resembles the identity near 0 and a small-activation tanh network trains much like a linear one. Sigmoidal units remain common in recurrent nets, probabilistic models, and some autoencoders, whose extra requirements rule out piecewise-linear units.

§6.3.3 Other hidden units — and a cautionary tale

Many differentiable functions work fine; new types are published only when clearly better (the authors got under 1% MNIST error with $g = \cos(Wx+b)$). Notable ones: linear/identity hidden units (factor a layer low-rank, $\boldsymbol{h} = g(\boldsymbol{V}\boldsymbol{U}^\top\boldsymbol{x}+\boldsymbol{b})$ with $(n+p)q < np$ parameters), softmax as a memory switch, RBF (saturates to 0, hard to optimize), hard tanh, and — a warning — softplus: smooth and differentiable everywhere, yet empirically worse than ReLU. “The performance of hidden unit types can be very counterintuitive”; smoothness is no guarantee.

§6.4 Architecture · §6.4.1 Universal approximation and depth

Architecture = how many units and how they connect. The common chain arranges layers $\boldsymbol{h}^{(k)} = g^{(k)}(\boldsymbol{W}^{(k)\top}\boldsymbol{h}^{(k-1)} + \boldsymbol{b}^{(k)})$; the main choices are depth and per-layer width. Remarkably, one hidden layer already suffices to represent anything:

So why go deep? Because depth buys exponential efficiency:

There is also a statistical reason. Choosing a deep model encodes a prior: that the target is a composition of simpler functions (a representation built of simpler representations — corners from edges) or a multi-step program (each step uses the previous step’s output). Empirically, greater depth generalizes better across many tasks (figs 6.6–6.7 — and crucially, adding depth beats adding the same number of parameters as width). Depth expresses a genuinely useful prior over functions.

Next: the rest of architecture (skip/sparse connections) and the algorithm that computes all these gradients — back-propagation.