§10.3–10.6Bidirectional · Encoder–Decoder · Deep · Recursive Nets

§10.3 Bidirectional RNNs

Every RNN so far is causal: the state at step $t$ sees only the past $x^{(1)}, \dots, x^{(t)}$. But many tasks need the whole input to predict $y^{(t)}$. In speech recognition, the right phoneme for the current sound can depend on the next few phonemes (co-articulation) — even the next few words. If two readings of a word are both acoustically plausible, you may have to look far ahead and behind to disambiguate.

Bidirectional RNNs bidirectional rnn An RNN combining a forward sub-RNN (state h) and a backward sub-RNN (state g), so the output o^(t) depends on the whole input sequence — both past and future — useful for speech and handwriting. defined in ch. 10 — open in glossary (Schuster & Paliwal 1997) solve this by combining a sub-RNN that runs forward (state $h^{(t)}$) with one that runs backward (state $g^{(t)}$). The output $o^{(t)}$ then draws on a summary of the past (via $h$) and the future (via $g$) — without any fixed-size look-ahead window:

The idea extends naturally to 2-D data such as images: use four RNNs, one each going up, down, left, and right. At each grid point, the output captures mostly local information but can also depend on long-range inputs. Versus a convolutional network this is more expensive, but it allows long-range lateral interactions across a feature map (the bottom-up input is itself a convolution).

§10.4 Encoder–decoder sequence-to-sequence architectures

We have now seen an RNN map a sequence → fixed vector (fig 10.5), a fixed vector → sequence (fig 10.9), and a sequence → equal-length sequence (figs 10.3/10.10/10.11). The missing case — the one behind machine translation and question answering — is a sequence → sequence of a different length.

The encoder–decoder encoder-decoder (sequence-to-sequence) An architecture where an encoder RNN reads the input to a context C (typically its final state) and a decoder RNN generates the output conditioned on C. The input and output lengths may differ — the key advance for machine translation. defined in ch. 10 — open in glossary (a.k.a. sequence-to-sequence) architecture (Cho 2014; Sutskever 2014) is disarmingly simple: an encoder RNN reads the input and emits a context context vector The fixed-length summary C that an encoder emits from an input sequence and passes to a decoder. When too small to capture a long input, it motivates the attention mechanism. defined in ch. 10 — open in glossary $C$ (usually its final hidden state); a decoder RNN, conditioned on $C$, generates the output. Slide the two lengths apart — nothing forces them equal:

The two RNNs are trained jointly to maximize the average of $\log P(y^{(1)}, \dots, y^{(n_y)} \mid x^{(1)}, \dots, x^{(n_x)})$ over all pairs. The encoder’s last state serves as $C$; there is no constraint that encoder and decoder share a hidden-layer size.

§10.5 Deep recurrent networks

Most RNN computation decomposes into three blocks: input→hidden, hidden→hidden, and hidden→output. In the workhorse RNN each block is a single weight matrix — a shallow transformation (one affine map + one nonlinearity). Would depth in each help? Experiments (Graves 2013; Pascanu 2014a) say yes:

Adding depth boosts capacity, but there is a cost: it lengthens the shortest path from a variable at step $t$ to one at step $t+1$. A single hidden layer in the state-to-state transition, for instance, doubles that path length versus the ordinary RNN — and shorter paths are easier to optimize. The mitigation (fig 10.13c) is skip connections skip connections through time Recurrent connections with a delay d>1 that link a unit to one d steps in the past, so gradients decay over τ/d rather than τ steps — one way to capture coarser (longer) time scales. defined in ch. 10 — open in glossary in the hidden-to-hidden path.

§10.6 Recursive neural networks

A recursive neural network recursive neural network A generalization of an RNN whose computational graph is a deep tree rather than a chain, reducing depth for a length-τ sequence from τ to O(log τ). Not abbreviated ‘RNN’ to avoid confusion with recurrent nets. defined in ch. 10 — open in glossary generalizes the RNN differently: its computational graph is a deep tree, not a chain (Pollack 1990). A variable-size sequence is folded up the tree to a fixed-size representation, with one shared set of weight matrices at every node. Drag $\tau$ to see why the tree shape matters:

For a sequence of length $\tau$, the depth — the number of composed nonlinear operations — drops from $\tau$ to $O(\log \tau)$, which can ease long-term dependency learning. The open question is how to structure the tree. Options: a data-independent shape (a balanced binary tree); an externally supplied shape (e.g. the parse tree of a sentence, Socher 2011/2013); or — ideally — a tree the learner discovers for each input. Node computations need not be the usual affine-plus-nonlinearity either: Socher (2013) uses tensor/bilinear operations to model relationships between concept embeddings.

Next: why long-range learning is hard in the first place — the challenge of long-term dependencies, echo state networks, and leaky units.