Parameter Initialization · Adaptive Learning Rates · AdaGrad · RMSProp

§8.4 Parameter initialization strategies

Deep-learning training is iterative, so it needs a starting point — and it is strongly affected by that choice, which can decide whether it converges at all, how fast, to what cost, and even how well it generalizes. Modern strategies are simple and heuristic. The one thing known with certainty: initialization must break symmetry. Two units with the same activation and the same inputs, given identical initial weights, are updated identically forever — so they stay redundant. Hence random initialization , drawn from a Gaussian or uniform distribution (the shape barely matters; the scale matters enormously).

Why scale is so delicate: each layer multiplies the activation’s standard deviation by an effective gain. Too small and the signal vanishes before it reaches the output; too large and it explodes. Slide the gain across an 8-layer chain:

Activation std across depth: each layer multiplies it by a gain g = √(fan-in)·(init scale). Only g ≈ 1 keeps the signal alive; g<1 vanishes, g>1 explodes

gain g = 1.00

std at layer 8 = g^8 = 1.00

stable — signal preserved

Each layer multiplies the activation std by an effective gain g = √(fan-in)·(init scale). Too small and the signal dies before it reaches the output; too large and it blows up. Xavier/Glorot init picks the scale so g ≈ 1 — and random init also breaks symmetry so units learn different things.

The classic fix is the normalized (Xavier–Glorot) initialization, which picks the scale from the layer’s fan-in $m$ and fan-out $n$ :

Term by term

W_{i,j} \sim U\!\left(-\sqrt{\tfrac{6}{m+n}},\ \sqrt{\tfrac{6}{m+n}}\right)

$m,\ n$	the layer's number of inputs and outputs — the scale shrinks as the layer gets wider, keeping each unit's total input controlled	fan-in/out
$\sqrt{6/(m+n)}$	the sweet-spot scale: a compromise that keeps BOTH the forward-activation variance and the backward-gradient variance roughly equal across layers	the balance
$\text{(linear assumption)}$	derived for a chain of matrix multiplies with no nonlinearities — yet works well in practice. Orthogonal or gain-tuned init (Saxe, Sussillo) can even train 1000 layers	the caveat

Weights are random; biases are (mostly) constants

Only the weights are randomized; biases are set to heuristic constants — usually 0, but with useful exceptions: set an output bias to match the marginal statistics of the target (solve softmax(b) = c); set a ReLU bias to 0.1 to avoid saturating it at initialization; set a gate unit’s bias so it starts open (the LSTM forget gate is initialized to 1). One more option, sparse initialization , gives each unit exactly $k$ nonzero weights so its total input is independent of fan-in without shrinking every weight — at the cost of a strong prior on those few large values. In practice the init scale is best treated as a per-layer hyperparameter, tuned near these theoretical values.

§8.5 Adaptive learning rates

The learning rate is the single hardest hyperparameter to set, and the cost is sensitive to some directions and flat in others. Momentum helps but adds another hyperparameter. A different idea: give each parameter its own learning rate and adapt it automatically. (The early delta-bar-delta heuristic did this for full-batch: grow a parameter’s rate while its partial derivative keeps its sign, shrink it when the sign flips.)

§8.5.1 AdaGrad

AdaGrad scales each parameter’s rate down by the square root of its accumulated squared gradients — so steep directions slow quickly while gently-sloped ones keep a large rate:

Algorithm 8.4 — AdaGrad (click lines)

1Require: global learning rate $\epsilon$ ; initial $\boldsymbol{\theta}$ ; small $\delta \approx 10^{-7}$ ⓘ
2initialize gradient accumulator $\boldsymbol{r} \leftarrow 0$
3while stopping criterion not met do
4compute gradient: $\boldsymbol{g} \leftarrow \frac{1}{m}\nabla_{\boldsymbol{\theta}}\sum_i L$
5accumulate squared gradient: $\boldsymbol{r} \leftarrow \boldsymbol{r} + \boldsymbol{g}\odot\boldsymbol{g}$ ⓘ
6compute update: $\Delta\boldsymbol{\theta} \leftarrow -\dfrac{\epsilon}{\delta + \sqrt{\boldsymbol{r}}}\odot\boldsymbol{g}$ ⓘ
7apply update: $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \Delta\boldsymbol{\theta}$

AdaGrad has nice convex guarantees, but for deep nets that full-history accumulation is a flaw: $\boldsymbol{r}$ grows from the very start, so the effective learning rate can shrink too much, too early, before reaching a good region.

§8.5.2 RMSProp

RMSProp fixes exactly that: replace the ever-growing sum with an exponentially-decayed moving average, so the distant past is forgotten:

Algorithm 8.5 — RMSProp (click lines)

1Require: global learning rate $\epsilon$ , decay rate $\rho$ ; initial $\boldsymbol{\theta}$ ; small $\delta \approx 10^{-6}$ ⓘ
2initialize accumulator $\boldsymbol{r} \leftarrow 0$
3while stopping criterion not met do
4compute gradient: $\boldsymbol{g} \leftarrow \frac{1}{m}\nabla_{\boldsymbol{\theta}}\sum_i L$
5accumulate (moving average): $\boldsymbol{r} \leftarrow \rho\boldsymbol{r} + (1-\rho)\,\boldsymbol{g}\odot\boldsymbol{g}$ ⓘ
6compute update: $\Delta\boldsymbol{\theta} = -\dfrac{\epsilon}{\sqrt{\delta + \boldsymbol{r}}}\odot\boldsymbol{g}$
7apply update: $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \Delta\boldsymbol{\theta}$

Because it discards ancient history, RMSProp behaves like AdaGrad restarted once the trajectory settles into a locally-convex bowl — so it converges rapidly there. It is empirically effective and one of the go-to optimizers for deep nets.

Next: Adam (momentum + RMSProp), how to choose an optimizer, and Newton’s method — Adam & Newton’s method.

Check yourself — initialization and adaptive learning rates

1.What is the one property initialization must have, and why?

2.In InitLab, why does a gain g slightly above 1 make the activation std explode across 8 layers, while g slightly below 1 makes it vanish?

3.What does the Xavier–Glorot scale √(6/(m+n)) accomplish?

4.AdaGrad scales each parameter's rate by 1/√r with r = Σ g⊙g over all history. What is its weakness for deep nets?

5.How does RMSProp differ from AdaGrad, and why does it help in the non-convex setting?

5 questions