§3.1The Transformer Architecture

In [Chapter 2 of the full book] we saw how to fine-tune and use a transformer transformer A neural architecture built on self-attention (no recurrence or convolution) that maps a sequence of token embeddings to a sequence of contextualized embeddings. defined in ch. 3 — open in glossary ; this chapter opens the hood and builds one from scratch in PyTorch. We start from the original 2017 architecture, assemble a working encoder piece by piece, peek at the decoder, and finish with a tour of the model zoo. A deep grasp of the internals isn’t strictly needed to apply transformers, but it is what lets you reason about their limits and adapt them to new domains.

Recap from Chapter 2 (assumed background). Text is first tokenized into sub-word units, each mapped to an integer input id; an embedding lookup table then turns each id into a dense vector. Everything below operates on those vectors.

The encoder–decoder architecture

The original Transformer was designed for sequence-to-sequence problems such as machine translation, and uses an encoder–decoder architecture encoder-decoder architecture The original transformer form: an encoder builds a representation of the input and a decoder generates an output sequence from it. Used for tasks like translation. defined in ch. 3 — open in glossary with two halves:

• The encoder encoder A stack of layers that turns input tokens into a sequence of contextualized embeddings (the hidden state / context). defined in ch. 3 — open in glossary converts an input sequence of tokens into a sequence of embedding vectors — the hidden state hidden state The sequence of embedding vectors the encoder produces for a sequence, also called the context. defined in ch. 3 — open in glossary or context.
• The decoder decoder A stack of layers that generates an output sequence one token at a time, autoregressively, using the encoder's output and its own past outputs. defined in ch. 3 — open in glossary uses that hidden state to iteratively generate an output sequence of tokens, one at a time.

Three things about the architecture are already visible:

1. Positions are injected on purpose. Because attention is order-blind (see §3.4), the token embeddings token embedding The fixed vector a token id maps to via a lookup table (768-dimensional in BERT), before any context is mixed in. defined in ch. 3 — open in glossary are summed with positional embeddings positional embedding A position-dependent vector added to token embeddings so the otherwise order-blind attention mechanism can model token order. defined in ch. 3 — open in glossary so the model knows where each token sits.
2. Both halves are stacks. The encoder is a stack of identical encoder layers (“blocks”), much like stacking convolutional layers in vision; the decoder has its own stack.
3. The encoder feeds the decoder. The encoder’s output is handed to every decoder layer, which then predicts the most probable next token — and that prediction is fed back in to produce the next one, until an end-of-sequence (EOS) token appears.

Watch the decoder generate

Step through translating “Time flies like an arrow” into German. Notice the encoder runs once, while the decoder loops — each predicted token becomes part of the next step’s input.

Zooming into the encoder

The encoder is a stack of identical layers, each of which takes a sequence of embeddings and passes them through two sublayers: a multi-head self-attention layer and a position-wise feed-forward layer. Crucially, the output embeddings have the same shape as the inputs — a layer’s job is to update each embedding with context (the word “apple” drifts toward “company” when “keynote” or “phone” sits nearby). That shape-preserving property is what lets us stack the layers.

Each sublayer is also wrapped in a skip connection and layer normalization — standard tricks for training deep networks that we add in §3.4. For now, the takeaway is the two-sublayer shape. To really understand the encoder we have to go deeper, starting with its beating heart: the self-attention layer.

Three flavors of transformer

Although the original design pairs an encoder with a decoder, each half was soon used on its own. Almost every model you’ll meet is one of three types.

• Encoder-only encoder-only Models (the BERT family) that use bidirectional attention to build rich representations; best for understanding tasks like classification, NER, and QA. defined in ch. 3 — open in glossary models (BERT and friends) build a rich representation using bidirectional attention bidirectional attention Attention in which a token's representation depends on both its left and right context. defined in ch. 3 — open in glossary — great for classification, NER, and extractive QA (Ch. 7).
• Decoder-only decoder-only Models (the GPT family) that use causal attention to predict the next token; best for text generation. defined in ch. 3 — open in glossary models (the GPT family) auto-complete text using causal (autoregressive) attention causal attention Autoregressive attention in which a token attends only to itself and earlier tokens, never to future ones. defined in ch. 3 — open in glossary , where each token sees only what came before it.
• Encoder–decoder encoder-decoder A model family with both an encoder and a decoder (e.g. T5, BART); maps one sequence to another, as in translation and summarization. defined in ch. 3 — open in glossary models (T5, BART) map one sequence to another — translation, summarization.

With the map in hand, let’s descend into the mechanism that makes all three work — self-attention.