Removing Connections · LSTM & Gated RNNs · Optimization

§10.9.3 Removing connections

A last multi-time-scale idea before gating. Organize the RNN’s state so information flows more easily over long distances at slower time scales — but do it by actively removing length-1 connections and replacing them with longer ones. This differs from skip connections: skips add edges, so a unit may still lean on its short-term connections; removing connections forces units onto a long time scale. Two ways: make groups of units leaky with different fixed time constants, or have groups update at different discrete frequencies (the “clockwork” RNN, Koutnik 2014).

§10.10 The LSTM and other gated RNNs

The most effective sequence models in practice are gated RNNs — the long short-term memory and the gated recurrent unit. Like leaky units, they create paths through time whose derivatives neither vanish nor explode. But where a leaky unit’s self-loop weight was a hand-chosen constant or a fixed parameter, a gated RNN makes it a learned weight that changes at every time step. A leaky unit can accumulate evidence — but once that evidence is used, you may want to forget it. Instead of deciding by hand when to clear the state, gated RNNs learn when to.

§10.10.1 The LSTM

The long short-term memory (Hochreiter & Schmidhuber 1997) introduces self-loops so gradients can flow for long durations — and its crucial refinement (Gers 2000) makes the self-loop weight gated (conditioned on context) rather than fixed. So even with fixed parameters, the time scale of integration changes dynamically, because the gates are output by the model itself. Play with the three gates and watch the cell state carry, forget, and read out over time:

Fig 10.16 (interactive) — the LSTM cell. Forget gate f weights the state self-loop, input gate g gates writing, output gate q gates reading. Set f≈1, g≈0 after the write to see the state held perfectly — the constant error carousel that carries gradients across many steps.

forget f = 0.95self-loop weightinput g = 1.00writeoutput q = 1.00read

f=0.95 sets the self-loop weight (memory ≈ 20 steps); g=1.00 gates how much of the input is written; q=1.00 gates how much of tanh(s) is read out as the output. Unlike a leaky unit, all three are learned and change every step.

The heart of the cell is the state $s_i^{(t)}$ , a linear self-loop whose weight is the forget gate, plus a gated input:

LSTM cell state and output (eqs 10.41, 10.43)

s_i^{(t)} = f_i^{(t)}\, s_i^{(t-1)} + g_i^{(t)}\,\sigma\!\Big(b_i + \textstyle\sum_j U_{i,j} x_j^{(t)} + \sum_j W_{i,j} h_j^{(t-1)}\Big), \qquad h_i^{(t)} = \tanh\!\big(s_i^{(t)}\big)\, q_i^{(t)}

$s_i^{(t)}$	the cell's internal state — a LINEAR self-loop, the memory along which gradients flow	per-cell scalar
$f_i^{(t)}\, s_i^{(t-1)}$	FORGET term: keep this fraction of the old state (f near 1 → remember for a long time)	gated retention
$g_i^{(t)}\,\sigma(\cdots)$	INPUT term: write this fraction of a fresh candidate value (an ordinary neuron's output) into the state	gated write
$h_i^{(t)} = \tanh(s_i^{(t)})\, q_i^{(t)}$	OUTPUT: the state, squashed by tanh, then gated by the output gate q	gated read

Each of the three gates is its own sigmoid unit — a value in $(0,1)$ acting as an input-dependent valve, with its own biases, input weights and recurrent weights:

The three gates — each a sigmoid valve (eqs 10.40, 10.42, 10.44)

f_i^{(t)} = \sigma\Big(b_i^{f} + \textstyle\sum_j U_{i,j}^{f} x_j^{(t)} + \sum_j W_{i,j}^{f} h_j^{(t-1)}\Big), \quad g_i^{(t)} = \sigma\big(\cdots^{\,g}\big), \quad q_i^{(t)} = \sigma\big(\cdots^{\,o}\big)

$\sigma(\cdot)$	every gate is a sigmoid → a value in (0,1): a soft, differentiable valve	gate value
$f_i^{(t)}$	FORGET gate — how much of the old cell state to keep (the self-loop weight)	(0,1)
$g_i^{(t)}$	INPUT (external input) gate — how much of the new candidate to accumulate	(0,1)
$q_i^{(t)}$	OUTPUT gate — how much of tanh(state) to expose as the cell output	(0,1)
$b^f, U^f, W^f$	each gate has its OWN parameters; a variant also feeds the cell state s into the gates ("peephole")	per-gate weights

LSTMs have proven extremely successful — handwriting recognition and generation, speech recognition, machine translation, image captioning, parsing — first on artificial long-term-dependency benchmarks, then on state-of-the-art real tasks.

§10.10.2 Other gated RNNs — the GRU

Which pieces of the LSTM are actually necessary? The gated recurrent unit (Cho 2014) is the main answer. Its key simplification: a single gate simultaneously controls both the forgetting factor and the update decision.

GRU update (eqs 10.45–10.47)

h_i^{(t)} = u_i^{(t-1)} h_i^{(t-1)} + \big(1 - u_i^{(t-1)}\big)\,\sigma\!\Big(b_i + \textstyle\sum_j U_{i,j} x_j^{(t-1)} + \sum_j W_{i,j}\, r_j^{(t-1)} h_j^{(t-1)}\Big)

$u_i^{(t-1)}$	UPDATE gate — a conditional leaky integrator: u→1 COPIES the old state, u→0 REPLACES it with the new target (folds the LSTM's forget + input into one)	(0,1)
$(1-u)\,\sigma(\cdots)$	the new "target state" the unit moves toward when the update gate opens	candidate
$r_j^{(t-1)}$	RESET gate — selects which parts of the past state feed the candidate (r→0 ignores the old state), adding a nonlinear past↔future interaction	(0,1)
$h_i^{(t)}$	the GRU has no separate cell state s: the gated hidden unit is both memory and output	per-unit scalar

Many variants, no clear winner — but keep the forget gate

Countless gating variants have been tried (sharing gates, global × local gates…). Broad studies (Greff 2015; Jozefowicz 2015) found no variant that clearly beats both the LSTM and GRU across tasks. Two robust findings: the forget gate is crucial, and adding a bias of +1 to the LSTM forget gate (so it defaults to “remember”) makes the LSTM as strong as the best variants.

§10.11 Optimization for long-term dependencies

Given the vanishing/exploding problem, should we attack it with a better optimizer? One idea (Martens & Sutskever 2011): second derivatives may vanish at the same rate as first derivatives, so a second-order method — which roughly divides gradient by curvature — could keep the ratio stable. Second-order methods showed promise, but they are costly, need large minibatches, and are drawn to saddle points. Later work found simpler Nesterov momentum with careful initialization could match them. And both have largely been replaced by simply using SGD (even without momentum) on LSTMs.

The recurring lesson

It is often much easier to design a model that is easy to optimize than to design a more powerful optimization algorithm. The LSTM’s gated linear self-loop did more for long-term dependencies than any clever optimizer — a theme that repeats across deep learning.

Two more tools remain for the exploding side and beyond: gradient clipping, information-flow regularization, and explicit memory.

Check yourself — the LSTM and other gated RNNs

1.How does 'removing connections' differ from adding skip connections through time?

2.What is the core idea shared by all gated RNNs (LSTM, GRU)?

3.In an LSTM, what does the forget gate do, and why is making the self-loop weight gated (rather than fixed) important?

4.In the LSTMLab, you set the forget gate f≈1 and the input gate g≈0 just after a write. What happens to the cell state, and what is this called?

5.How does the gated recurrent unit (GRU) simplify the LSTM, and what did architecture studies conclude?

6.What is the current practical wisdom on optimizing RNNs for long-term dependencies?

6 questions