§6.2.1–6.2.2aCost Functions · Linear & Sigmoid Output Units

§6.2.1 Cost functions

Neural-network cost functions are, happily, almost identical to those of any parametric model. In most cases the model defines a distribution $p(y \mid \boldsymbol{x}; \boldsymbol{\theta})$ and we apply maximum likelihood — i.e. minimize the cross-entropy cross-entropy cost J = −E_{x,y∼p̂} log p_model(y|x): the default neural-net cost from maximum likelihood; its log undoes the exp of saturating output units. defined in ch. 6 — open in glossary between data and model:

One quirk: cross-entropy usually has no minimum for real models. A discrete model like logistic regression can approach probability 0 or 1 but never reach it; a real-valued model that controls its output variance can drive the density — and thus the negative log-likelihood — toward $-\infty$. Regularization (chapter 7) removes this unbounded reward.

§6.2.1.2 Learning conditional statistics

Sometimes we want just one statistic of $y$, not the whole distribution. With a powerful enough network we can treat the cost as a functional functional A mapping from functions to real numbers; viewing the cost as a functional lets calculus of variations show which statistic each loss recovers. defined in ch. 6 — open in glossary (a map from functions to reals) and use calculus of variations to see which statistic each loss recovers. Squared error recovers the mean; the mean absolute error mean absolute error The L¹ loss E‖y − f(x)‖₁; its minimizer predicts the MEDIAN of y|x (MSE predicts the mean). defined in ch. 6 — open in glossary recovers the median:

§6.2.2 Output units

The choice of cost is tightly coupled to the output unit: the output representation fixes the form of the cross-entropy. (Any output unit can also serve as a hidden unit — §6.3 revisits them.) Throughout, suppose the network provides hidden features $\boldsymbol{h} = f(\boldsymbol{x}; \boldsymbol{\theta})$; the output layer transforms them to complete the task.

§6.2.2.1 Linear units — for Gaussian outputs

A linear output unit linear output unit An output unit that is a pure affine map ŷ = Wᵀh + b (no nonlinearity); used as the mean of a conditional Gaussian, so ML = MSE. Does not saturate. defined in ch. 6 — open in glossary is just an affine map with no nonlinearity: $\hat{\boldsymbol{y}} = \boldsymbol{W}^\top\boldsymbol{h} + \boldsymbol{b}$. Used as the mean of a conditional Gaussian $p(y \mid \boldsymbol{x}) = \mathcal{N}(y; \hat{\boldsymbol{y}}, \boldsymbol{I})$, maximizing log-likelihood = minimizing MSE. Because linear units do not saturate, they pose little trouble for gradient-based optimizers.

§6.2.2.2 Sigmoid units — for Bernoulli outputs

For a binary $y$, the net must predict $P(y = 1 \mid \boldsymbol{x}) \in [0,1]$. The tempting fix — threshold a linear unit to [0,1] — is a trap: any time the linear value strays outside the unit interval, its gradient is 0, and learning stalls. The sigmoid output unit sigmoid output unit ŷ = σ(wᵀh + b) for a Bernoulli output; paired with NLL loss J = ζ((1−2y)z) that saturates only when the model is already correct. defined in ch. 6 — open in glossary instead guarantees a strong gradient whenever the answer is wrong: $\hat{y} = \sigma(\boldsymbol{w}^\top\boldsymbol{h} + b)$. Motivate it by building an unnormalized distribution whose log is linear in the score $z = \boldsymbol{w}^\top\boldsymbol{h} + b$ — the logit logit The pre-sigmoid/pre-softmax score z = log P̃(y): an unnormalized log-probability that the output activation exponentiates and normalizes. defined in ch. 6 — open in glossary :

The payoff is in the loss. With maximum likelihood, $J = -\log P(y \mid \boldsymbol{x})$, and the log undoes the exp:

See it directly — the NLL grows without bound when wrong (living gradient), while MSE flattens on the wrong side (dead gradient):

Next: the multiclass generalization (softmax) and richer output types (mixture density networks): softmax and beyond.