Chapter 2 ended with a tensor of input embeddings — numbers the model can multiply. This chapter builds the machinery that reads them: the attention mechanism attention mechanism Lets a model directly access and weight all input positions when building each representation, instead of relying on one compressed hidden state. defined in ch. 3 — open in glossary , step 2 of Stage 1 in the book’s roadmap. Before writing any attention code, this unit answers the question the whole chapter hangs on: what problem does attention solve that older architectures could not?
The chapter climbs four rungs, each one a working PyTorch module that the next rung extends. You are at the very beginning of this ladder:
3.1 The problem with modeling long sequences
Machine translation makes the core difficulty concrete. Suppose we translate the German sentence “Kannst du mir helfen diesen Satz zu uebersetzen” into English. Mapping each word to its English counterpart in place produces garbage — because German and English grammar order words differently, some output words need information from earlier or later parts of the source sentence:
To cope with this, pre-transformer systems used a deep neural network with two submodules — an encoder encoder The transformer half that maps an input sequence into context-carrying embedding vectors. BERT is encoder-only. defined in ch. 1 — open in glossary that first reads in and processes the entire input text, and a decoder decoder The transformer half that generates output tokens one at a time. GPT is decoder-only; this book builds a decoder-only model. defined in ch. 1 — open in glossary that then produces the translated text. The classic realization was the encoder–decoder RNN (recurrent neural network). Step through how it processes our sentence — and where it breaks:
The encoder reads one token at a time
An RNN processes the input sequentially. It reads "Kannst" and folds it into a hidden state h₁ — a fixed-size vector summarizing everything seen so far.
An encoder–decoder RNN translating 'Kannst du mir …'. Step through to see why one memory cell becomes a bottleneck.
Key idea — the bottleneck
The encoder–decoder RNN cannot directly access earlier hidden states from the encoder during decoding. It relies solely on the current hidden state — one fixed-size vector that must encapsulate all relevant information about the input. For long or complex sentences, where dependencies span long distances, this single-vector bottleneck loses context.3.2 Capturing data dependencies with attention mechanisms
The fix, historically, arrived in two moves. First, researchers added an attention mechanism to the RNN translator (the Bahdanau attention of 2014): at every output step, the decoder gets a direct, weighted connection to all input positions — no more squeezing everything through the final hidden state. Step through what changes:
Generating "Can": all inputs visible, weighted
The decoder produces its first word with a direct view of EVERY input token. The connection widths are attention weights: "Kannst" (German for "can") gets the strongest one. No information had to survive a bottleneck.
An RNN decoder WITH attention: at each output step, dotted connections reach every input token; line width = how important that input is for the word being generated (weights are illustrative).
The second move (the 2017 transformer) was more radical: keep the idea of weighted access to everything, but drop the RNN entirely. Self-attention self-attention A mechanism that lets each token weigh the relevance of every other token in the sequence. It is the core component of the transformer. defined in ch. 1 — open in glossary is a mechanism that allows each position in the input sequence to attend to all positions in the same sequence when computing its representation. The “self” is the contrast with the translator above: there, attention connected two different sequences (German input ↔ English output); in self-attention, one sequence relates its own parts to each other — how relevant is every other word of this sentence to this word?
A brief history of getting rid of the bottleneck
Encoder–decoder RNNs (seq2seq, 2014) hit the fixed-vector bottleneck almost immediately. Bahdanau et al. (2014) bolted attention onto the RNN so the decoder could look back at all encoder states. “Attention Is All You Need” (2017) then showed the RNN itself was unnecessary — self-attention alone, in the transformer transformer The neural architecture behind modern LLMs — an encoder and/or decoder built around attention — introduced in "Attention Is All You Need" (2017). defined in ch. 1 — open in glossary architecture, models the dependencies better and trains in parallel. That architecture powers the GPT gpt A decoder-only transformer trained by next-word (autoregressive) prediction; strong at text generation. The model built in this book. defined in ch. 1 — open in glossary series this book builds.Self-attention is the key component of contemporary transformer-based LLMs. Inside the GPT-like decoder-only stack, it is exactly the piece this chapter constructs — everything else around it is either last chapter’s preprocessing or next chapter’s architecture:
Self-attention vs. “traditional” attention
Both compute importance weights, but over different things. Traditional (cross) attention relates elements of two different sequences — the RNN translator above attends from the output sentence to the input sentence. Self-attention relates positions within a single sequence to each other — words of one sentence (or even pixels of one image). GPT uses self-attention: when processing “journey”, it asks how relevant “Your”, “starts”, “with”… are to it. The next section computes exactly that, by hand.What comes next: §3.3 strips attention down to its bare minimum — no trainable weights at all, just dot products and a softmax — and computes a full attention matrix for the six-token sentence “Your journey starts with one step” from chapter 2’s embeddings.
📝 Check yourself: why attention?
0 / 51.Why can't we translate a German sentence to English simply word by word?
2.In an encoder–decoder RNN, what is the decoder's ONLY source of information about the input sentence?
3.What does adding an attention mechanism to the RNN translator (Bahdanau attention) change?
4.Predict: in the attention walkthrough above, when the decoder is generating the output word "you", which input token should get the strongest (widest) connection?
5.What does the "self" in self-attention refer to?