§10.7–10.9.2Long-Term Dependencies · Echo State Networks · Leaky Units

§10.7 The challenge of long-term dependencies

We met this obstacle in §8.2.5; now we see it in its natural home. The problem: gradients propagated over many time steps tend to either vanish (almost always) or explode (rarely, but catastrophically). An RNN composes the same function once per step, and that composition is a product of many Jacobians — so even a perfectly stable RNN gives a long-term interaction exponentially smaller weight than a short-term one. Composing many linear-tanh layers produces a wildly nonlinear map (fig 10.15) whose derivatives are mostly tiny, with rare large spikes.

Strip away the nonlinearity and the input, and the recurrence becomes pure repeated matrix multiplication — the power method:

The magnitude after $t$ steps scales as the spectral radius raised to the power $t$. Drag $\rho$ across 1 to see the three regimes — decay, edge of stability, explosion — in both the magnitude curve and a vector spiralling in or out under repeated multiplication:

Contrast the scalar case: multiplying a weight $w$ by itself gives $w^t$, which vanishes or explodes. But a non-recurrent net with a different weight $w^{(t)}$ each step can be scaled (variance $\propto 1/n$) to keep the product well-behaved (Sussillo 2014) — the difficulty is special to sharing one $W$ across time.

§10.8 Echo state networks

The input weights ($x \to h$) and recurrent weights ($h \to h$) are the hardest parameters to learn. Echo state networks echo state network (esn) A recurrent network whose input and recurrent weights are fixed (not learned) so the hidden units form a rich reservoir of temporal features; only the output weights are trained, often via a convex criterion. defined in ch. 10 — open in glossary (and the spiking-neuron liquid state machines) sidestep the problem entirely: fix those weights so the hidden units form a rich reservoir of temporal features, and learn only the output weights. Because the readout is typically a linear regression from the state, its training criterion is convex and solvable reliably. This family is called reservoir computing reservoir computing The umbrella term (covering echo state networks and liquid state machines) for models that map a sequence into a fixed-length recurrent state — a fixed ‘reservoir’ of features — on which a simple linear predictor is trained. defined in ch. 10 — open in glossary .

How do you set the fixed weights? View the recurrent net as a dynamical system and place it near the edge of stability — make the eigenvalues of the state-to-state Jacobian near 1. Its spectral radius spectral radius The maximum absolute value among a matrix’s eigenvalues. For an RNN’s state-to-state Jacobian it governs whether small perturbations (and gradients) grow or shrink over time; reservoir methods tune it near the edge of stability. defined in ch. 10 — open in glossary (largest $|\lambda|$) governs whether a perturbation grows or shrinks: after $n$ back-prop steps a deviation of size $\delta$ becomes $\delta|\lambda|^n$ — explode if $|\lambda| > 1$, shrink if $|\lambda| < 1$ (rewind the widget above to see it). A contractive map ($|\lambda| < 1$) necessarily forgets the past at finite precision.

Crucially, a saturating tanh bounds the forward dynamics — the derivative approaches 0 on many steps and prevents explosion — so modern ESNs deliberately use a spectral radius much larger than 1 (around 3). The same trick even helps initialize a fully trainable RNN: start with spectral radius ≈ 1.2 plus the sparse initialization of §8.4, then train the recurrent weights normally.

§10.9 Leaky units and a spectrum of time scales

A different tack: build a model that operates at multiple time scales at once — some parts fine-grained (small details), others coarse (carrying information from the distant past efficiently). Three strategies appear: adding skip connections across time, using leaky units, and removing fine-scale connections (the last covered next).

§10.9.1 Skip connections through time

Add direct edges from the distant past to the present. Where an ordinary recurrent connection spans one step ($t \to t+1$), a connection with delay $d$ spans $d$ steps — so gradients decay over $\tau/d$ rather than $\tau$:

§10.9.2 Leaky units

Another way to get a product of derivatives near 1: a unit with a linear self-connection whose weight is near 1. Accumulate a running average $\mu^{(t)} \leftarrow \alpha\,\mu^{(t-1)} + (1-\alpha)\,v^{(t)}$ — the coefficient $\alpha$ is exactly such a self-loop. A leaky unit leaky unit A hidden unit with a linear self-connection µ^(t) ← αµ^(t-1) + (1-α)v^(t) acting as a running average. α near 1 remembers the distant past; α near 0 forgets quickly — a smooth, adaptable way to hold information over time. defined in ch. 10 — open in glossary is a hidden unit that behaves this way. Slide $\alpha$ to trade fast forgetting for long memory:

Where a skip connection with delay $d$ ties a unit to a value exactly $d$ steps back (an integer), the linear self-loop lets the same effect be adapted smoothly by tuning the real-valued $\alpha$. The time constants can be fixed (sampled once at initialization) or made free parameters and learned; leaky units at different time scales help capture long-term dependencies.

Next: forcing units onto coarse time scales, and the architecture that finally tames long-term dependencies — removing connections, and the LSTM & gated RNNs.