Gradient Clipping · Information Flow · Explicit Memory

§10.11.1 Clipping gradients

A recurrent net over many steps computes a strongly nonlinear function, and such functions have cliffs — wide, nearly-flat regions separated by tiny regions where the cost changes violently (§8.2.4, figs 8.3 & 10.17). The cliff is exponentially steep in the number of steps because the weight matrix is multiplied by itself once per step. The danger: a very large gradient makes an ordinary update throw the parameters far into a worse region, undoing much of the work done to get there. The gradient only describes an infinitesimal neighborhood; just outside it, the cost curves back up.

The long-standing fix is to clip the gradient. Drag the threshold to watch the raw gradient spike at the cliff get capped:

Fig 10.17 (interactive) — norm clipping. Along a descent path crossing a cliff the gradient norm spikes (dashed red); clipping caps it at the threshold v (solid green) so the update cannot catapult the parameters.

clip threshold v = 8

At the cliff the raw gradient norm reaches ≈ 45 (dashed). Norm clipping rescales any g with ‖g‖ > v to g·v/‖g‖, capping the update norm at 8 (solid) — same direction, bounded length, so the exploding gradient can't catapult the parameters across the ravine.

There are two variants. Element-wise clipping (Mikolov 2012) caps each component; norm clipping (Pascanu 2013) rescales the whole vector:

Norm clipping (eqs 10.48–10.49)

\text{if } \lVert g \rVert > v: \qquad g \leftarrow \frac{g\, v}{\lVert g \rVert}

$\lVert g \rVert > v$	only act when the gradient norm exceeds the threshold v (a hyperparameter)	condition
$\dfrac{g\, v}{\lVert g \rVert}$	rescale g to have norm exactly v — SAME direction, bounded LENGTH	the clip
$v$	the norm threshold; the update can never be longer than this, so an exploding gradient cannot fling the parameters away	scalar

Because norm clipping rescales all parameters jointly, the step still points along the true gradient — only its length is bounded; both variants behave similarly in practice. A few practical notes: even taking a random step of size $v$ when the gradient exceeds the threshold works almost as well (and if the gradient is Inf/NaN, a random step of size $v$ typically escapes the unstable spot). Per-minibatch norm clipping keeps each minibatch’s direction, but averaging norm-clipped minibatch gradients is not the same as clipping the true gradient — so, unlike unbiased SGD, it introduces a heuristic bias that is empirically useful. One can also clip the back-propagated (hidden-unit) gradient directly (Graves 2013).

§10.11.2 Regularizing to encourage information flow

Clipping tames explosion but does nothing for vanishing. To keep long-term gradients alive without the machinery of an LSTM, one can regularize the parameters to preserve gradient magnitude — encouraging the back-propagated $\nabla_{h^{(t)}} L$ to stay as large after one step back as before:

Information-flow regularizer (eq 10.52)

\Omega = \sum_t \left( \frac{\big\lVert (\nabla_{h^{(t)}} L)\, \frac{\partial h^{(t)}}{\partial h^{(t-1)}} \big\rVert}{\lVert \nabla_{h^{(t)}} L \rVert} - 1 \right)^{2}

$\frac{\partial h^{(t)}}{\partial h^{(t-1)}}$	the Jacobian that carries the back-propagated gradient one step further back in time	\|h\|×\|h\|
$\text{the ratio}$	how much the gradient magnitude changes over one backward step — we want it ≈ 1 (neither shrink nor grow)	scalar
$(\cdots - 1)^2$	penalize any deviation from magnitude-preservation, summed over all steps t	penalty

Treating the back-propagated $\nabla_{h^{(t)}} L$ as a constant makes this cheap to differentiate. Combined with norm clipping (which keeps the now-encouraged gradients from exploding), it considerably increases the span of dependencies an RNN can learn — though it is not as effective as an LSTM when data is abundant.

§10.12 Explicit memory

Some knowledge is implicit — subconscious, hard to verbalize (how to walk; how a cat differs from a dog). Other knowledge is explicit — declarative facts (“a cat is an animal”; “the meeting is at 3 PM in room 141”). Neural networks excel at implicit knowledge but struggle to memorize facts: SGD needs many presentations to store an input, and even then imprecisely. They lack a working memory for holding and manipulating task-relevant facts.

Explicit memory components address this. Memory networks (Weston 2014) add a set of memory cells reached by an addressing mechanism (originally needing supervision on how to use them). The neural Turing machine (Graves 2014) learns to read and write without that supervision, using differentiable soft addressing — so the whole system trains end to end:

Fig 10.18 — a network with explicit memory. A task network (a controller, often recurrent) learns to read from and write to specific addresses in a bank of memory cells via the reading and writing mechanisms (bold arrows).

The trick that makes memory differentiable: exact integer addresses are hard to optimize, so an NTM reads and writes many cells at once — a softmax-weighted average, concentrated (but never exactly one-hot) on a few cells. Because the weights have non-zero derivatives, gradient descent can learn what to access. Drag the query and sharpness to see soft addressing in action:

Fig (interactive) — content-based soft addressing. A query is compared to each cell's key; a softmax gives read weights; the read is Σ wᵢ·vᵢ. Higher sharpness → more focused (near one-hot); it stays differentiable throughout. This is exactly an attention read.

query key q = 3.0sharpness = 1.5

cell Akey 1 · v=12

cell Bkey 2 · v=34

0.15

cell Ckey 3 · v=56

0.69

cell Dkey 4 · v=78

0.15

cell Ekey 5 · v=90

read = Σ wᵢ·vᵢ = 56.0query q matched most strongly to the nearest key

The softmax focuses the read on cells whose key matches the query, while staying soft. Reading a weighted average of many cells (not one integer address) keeps the operation differentiable, so gradient descent can train what to read. This content-based soft addressing is identical in form to the attention mechanism.

Cells hold vectors (not scalars), which unlocks two addressing modes:

Two ways to address memory — click a cell for detail.
	How it selects a cell	Read instruction	Most useful when
Content-based	by how well the cell content matches a query pattern	“retrieve the song with the chorus ‘yellow submarine’”	objects to retrieve are large
Location-based	by slot number, ignoring the content	“retrieve the song in slot 347”	even when cells are small

Dotted-underlined cells have explanations — click one.

If a cell’s content is copied (not forgotten) across steps, information — and its gradient — propagates across long durations without vanishing or exploding. Alternatively, one can interpret the weights as probabilities and stochastically read just one cell (Zaremba & Sutskever 2015), though such hard, discrete-decision models are harder to train than soft ones.

Soft addressing is attention

Whether soft (differentiable) or stochastic-and-hard, the mechanism for choosing an address is identical in form to the attention mechanism introduced for machine translation (§12.4.5). At each step the focus can move to a completely different location. Recurrent networks extend deep learning to sequential data — the last major tool in the toolbox. From here, the book turns to how to choose and apply these tools to real-world tasks.

That completes Chapter 10. Next, Part II closes with practical methodology and applications — see the chapter map.

Check yourself — gradient clipping, information flow, and explicit memory

1.Why do 'cliffs' in an RNN's loss landscape cause trouble, and how does gradient clipping help?

2.What does norm gradient clipping do (eqs 10.48–10.49)?

3.Which statement about clipping variants is TRUE?

4.Gradient clipping handles explosion. What is the 'regularizing to encourage information flow' idea for the vanishing side?

5.Why add EXPLICIT memory (memory networks, the neural Turing machine) to a network?

6.In the MemoryAddressLab, reading takes a softmax-weighted average over cells rather than one integer address. Why, and what is content- vs location-based addressing?

6 questions