Neural Machine Translation · Attention · Historical Perspective

§12.4.5 Neural machine translation

Machine translation reads a sentence in one language and emits an equivalent one in another. Classic systems have a proposal component (suggesting candidate translations, many ungrammatical — “apple red”) and a language model that scores them (“red apple” > “apple red”). Neural nets entered first as that language model (replacing an n-gram scorer), and — since translation produces an output given an input — it’s natural to make the model conditional (§6.2.1.1).

The flexible solution is the encoder–decoder (fig 12.5): an encoder (RNN or CNN) reads the source into a context $C$ — a representation of the whole sentence — and a decoder RNN generates the target from $C$ . Because the two lengths can differ, this fits translation (and even cross-modal tasks like image captioning: an image encoder → a text decoder). It’s the same machinery we built in §10.4:

Fig 12.5 (interactive) — the encoder-decoder for translation: an encoder reads the source sentence into a context C, a decoder emits the target. The two lengths need not match (French → English).

input length n_x = 4output length n_y = 3

The encoder compresses 4 inputs into the single fixed context C; the decoder unrolls it into 3 outputs. Because C sits between them, the two lengths need not match (here nₓ=4 ≠ nᵧ=3) — the advance over same-length RNNs.

§12.4.5.1 Attention

Squeezing a 60-word sentence into one fixed vector $C$ is hard. Attention (Bahdanau 2015, fig 12.6) instead reads the whole sentence, then produces each output word while focusing on a different part of the input. It is, essentially, a weighted average:

The attention mechanism (fig 12.6)

c = \sum_t \alpha^{(t)}\, h^{(t)}, \qquad \alpha = \operatorname{softmax}(\text{relevance scores}), \qquad \sum_t \alpha^{(t)} = 1

$h^{(t)}$	feature vectors — hidden units (or even raw inputs), one per source position	the memory
$\alpha^{(t)}$	attention weights in [0,1], produced by the model as a softmax over relevance scores — usually concentrated on one position	weights
$c = \sum_t \alpha^{(t)} h^{(t)}$	the context: a weighted average that approximates reading one specific h⁽ᵗ⁾ precisely	the read
$\text{differentiable}$	unlike hard indexing (which gradient descent can't train), this smooth weighted average CAN be optimized	key property

An attention-based system has three components: (1) a reader turning raw data into a feature vector per position, (2) that list of feature vectors as a memory of facts, and (3) a process that exploits the memory — attending to one (or a few) elements at each step to perform the task. This is identical in form to the soft memory addressing of a neural Turing machine (§10.12): a softmax-weighted read over a set of elements. Move the query and sharpness to see it:

Fig 12.6 (interactive) — attention as soft addressing. A query scores each source position; a softmax gives the weights α⁽ᵗ⁾; the read is Σ α⁽ᵗ⁾·h⁽ᵗ⁾. Sharper weights focus on one word — exactly the alignment attention learns in translation.

query key q = 3.0sharpness = 1.5

cell Akey 1 · v=12

cell Bkey 2 · v=34

0.15

cell Ckey 3 · v=56

0.69

cell Dkey 4 · v=78

0.15

cell Ekey 5 · v=90

read = Σ wᵢ·vᵢ = 56.0query q matched most strongly to the nearest key

The softmax focuses the read on cells whose key matches the query, while staying soft. Reading a weighted average of many cells (not one integer address) keeps the operation differentiable, so gradient descent can train what to read. This content-based soft addressing is identical in form to the attention mechanism.

Aligning source and target words this way also lets a model learn a translation matrix between the two languages’ word embeddings, lowering alignment error versus phrase-table counts.

§12.4.6 Historical perspective

Distributed representations for language have a long history:

From symbols to word embeddings — the arc of distributed representations in NLP.

Representations of symbols (Rumelhart 1986, learning family relations like (Colin, Mother, Victoria)) gave way to word embeddings (Deerwester 1990, via SVD; later neural). Some early NLP models worked at the character level; Bengio (2001) returned focus to words with neural language models that produce interpretable embeddings, scaling from a handful of symbols to millions of words. These ideas spread to parsing, tagging, and semantic role labeling — often via multi-task architectures sharing one embedding — and t-SNE made 2-D embedding visualizations a standard analysis tool.

§12.5 Other applications

Beyond vision, speech, and core NLP, deep learning powers recommender systems, knowledge representation, and reasoning — the subject of the final section: recommender systems and knowledge representation.

Check yourself — neural machine translation and attention

1.How did neural networks first enter machine translation, and how does an encoder-decoder handle it?

2.What is the attention mechanism, essentially? (See the interactive.)

3.What three components make up an attention-based system?

4.Why is attention preferable to encoding a long sentence into a single fixed context vector?

5.The book notes attention is 'identical in form' to another mechanism from chapter 10. Which?

6.How did distributed representations for language evolve historically?

6 questions