Softmax Units · Other Output Types · Hidden Units Intro

§6.2.2.3 Softmax units — for multinoulli outputs

To represent a distribution over $n$ classes, generalize the sigmoid to the softmax output unit . A linear layer produces unnormalized log-probabilities $\boldsymbol{z} = \boldsymbol{W}^\top\boldsymbol{h} + \boldsymbol{b}$ ( $z_i = \log \tilde P(y = i \mid \boldsymbol{x})$ ), then softmax exponentiates and normalizes:

Term by term

\mathrm{softmax}(\boldsymbol{z})_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}, \qquad \log\mathrm{softmax}(\boldsymbol{z})_i = z_i - \log\sum_j \exp(z_j)

$z_i$	the first term of the LOG-softmax feeds the cost DIRECTLY and cannot saturate — so learning always has a signal to push zᵢ up	direct term
$\log\sum_j \exp(z_j) \approx \max_j z_j$	pushes ALL logits down; roughly the max, so the NLL strongly penalizes the most-active INCORRECT class	competition term
$\text{result}$	unregularized ML drives softmax to the empirical class FREQUENCIES (consistent estimator, §5.4.5)	what it learns

Slide the logits and watch the competition — raising one class’s logit lowers the others’ probabilities (they must sum to 1), and the correct class’s cost stays responsive:

Softmax over three classes — logits → probabilities, with the log-softmax cost split into its two terms

z12.0z20.5z3-1.0correct class

cost = −log softmax(z)₁
= −(z₁ − log Σ e^z)
= −(2.00 − 2.24) = 0.241

The first term z₁ feeds the cost directly and can never saturate, so learning always has a signal — even when the second term becomes tiny.

Competition, saturation, and a name

Softmax is a winner-take-all / lateral-inhibition mechanism (outputs sum to 1, so one unit rising forces others down — like cortical neurons). It saturates when logit differences grow extreme; only losses that use a log to undo the exp (like the NLL) keep learning through that — squared error fails badly on softmax outputs. The name misleads: it softens arg max (a differentiable one-hot), not max; the soft max itself is $\mathrm{softmax}(\boldsymbol{z})^\top\boldsymbol{z}$ .

Two more facts. Shift invariance: $\mathrm{softmax}(\boldsymbol{z}) = \mathrm{softmax}(\boldsymbol{z} - \max_i z_i)$ — the numerically stable form you met in §4.1’s stabilizer. Overparametrization: the $n$ outputs must sum to 1, so only $n-1$ are free; fixing $z_n = 0$ gives the restricted form, and the sigmoid is exactly softmax with a 2-D $z$ and $z_1 = 0$ . In practice the overparametrized version is simpler and just as good.

§6.2.2.4 Other output types

The general recipe: let the network output $\boldsymbol{\omega} = f(\boldsymbol{x}; \boldsymbol{\theta})$ be the parameters of a distribution over $y$ , and use the loss $-\log p(y; \boldsymbol{\omega}(\boldsymbol{x}))$ . This covers a lot:

Learning the variance → a heteroscedastic model (the output variance depends on $\boldsymbol{x}$ ). Use precision $\boldsymbol{\beta} = \zeta(\boldsymbol{a})$ (softplus for positivity) — the log-likelihood then involves only multiply/add/log, all well-behaved, whereas variance would need division (steep near 0) and standard deviation would need division and squaring (vanishing gradient near 0). Any covariance must stay positive definite ( $\boldsymbol{\Sigma} = \boldsymbol{B}\boldsymbol{B}^\top$ ; full is $O(d^3)$ , so diagonal is the norm).
Multimodal regression (several valid $y$ for one $\boldsymbol{x}$ ) → a Gaussian-mixture output = a mixture density network :

Term by term

p(y \mid \boldsymbol{x}) = \sum_{i=1}^n p(c = i \mid \boldsymbol{x})\, \mathcal{N}\big(y;\, \boldsymbol{\mu}^{(i)}(\boldsymbol{x}),\, \boldsymbol{\Sigma}^{(i)}(\boldsymbol{x})\big)

$p(c=i \mid \boldsymbol{x})$	mixture weights over the latent component c — a SOFTMAX output (positive, sums to 1)	multinoulli
$\boldsymbol{\mu}^{(i)}(\boldsymbol{x})$	per-component means — unconstrained (usually no output nonlinearity); ML weights each example by the probability that component produced it	n×d means
$\boldsymbol{\Sigma}^{(i)}(\boldsymbol{x})$	per-component covariances — typically diagonal; gradients can be unstable (division by small variances) → clip gradients	n covariances

The three groups of outputs — mixing weights, means, covariances — all vary nonlinearly with $\boldsymbol{x}$ . This is exactly a conditional, input-dependent version of the Gaussian mixtures of §3.9.6:

A mixture output can place mass on several peaks at once (the fixed-input view of an MDN, fig 6.4)

α0.50μ₁-1.20μ₂1.20σ0.50

P(x) = α·N(x; μ₁, σ²) + (1−α)·N(x; μ₂, σ²): the prior α = P(c=1) = 0.50 picks the component; merge the means and the modes fuse

Even richer outputs (sequences of characters forming a sentence) push beyond this chapter — recurrent nets (ch. 10) and Part III handle those.

§6.3 Hidden units — the decision unique to deep networks

Everything so far (cost, output units) is shared with linear models. The genuinely new choice is the hidden unit — and it is the least settled: an active research area with few definitive principles. The working guidance:

ReLU is an excellent default. When in doubt, use it.
Choose by experiment. Intuit a unit type, train, evaluate on the validation set — you generally cannot predict the winner in advance.
Most hidden units are the same shape: an affine map $\boldsymbol{z} = \boldsymbol{W}^\top\boldsymbol{x} + \boldsymbol{b}$ then an element-wise nonlinearity $g(\boldsymbol{z})$ — they differ only in $g$ .

Aside — the non-differentiable point (ReLU@0) is fine

ReLU isn’t differentiable at $z = 0$ — which seems to disqualify it for gradient methods. In practice it’s a non-issue: training never actually reaches a point where the gradient is exactly 0 (it merely drives the cost low, fig 4.3), so minima at non-differentiable points are acceptable. Each such unit has a well-defined LEFT and RIGHT derivative (for ReLU@0: left = 0, right = 1); software just returns one of them — heuristically justified because the “0” input was probably a small value rounded to 0 anyway.

The catalog of hidden units — ReLU and its generalizations, the maxout unit, sigmoid/tanh, and the surprising failures — is next, along with how deep to stack them.

Check yourself — softmax, MDNs, hidden units

1.In log softmax(z)_i = z_i − log Σ exp(z_j), why does the first term keep learning alive?

2.In SoftmaxLab you raise z₁. What happens to the other classes' probabilities, and what property is this?

3.The name 'softmax' is a bit misleading. What does it actually soften?

4.Why parametrize a learned Gaussian output by PRECISION β = ζ(a) rather than by variance or standard deviation?

5.ReLU is not differentiable at z = 0. Why is that acceptable for gradient-based training?

5 questions