We’ve now dissected the original Transformer end to end. To close the chapter, let’s zoom back out: the encoder, decoder, and encoder-decoder designs each triggered a Cambrian explosion of models. Almost all of them still fall into one of those three families.
The transformer tree of life
Figure 3-8 highlights a handful of architectural milestones. Explore it — click any model to see its contribution:
Click any model to see what made it notable. Each branch reads top-to-bottom in roughly chronological order — the “Cambrian explosion” of transformer models.
The encoder branch
The first encoder-only encoder-only Models (the BERT family) that use bidirectional attention to build rich representations; best for understanding tasks like classification, NER, and QA. defined in ch. 3 — open in glossary model, BERT, shattered the GLUE natural-language-understanding ( NLU nlu Natural language understanding — tasks like classification, NER, and question answering where the model reads and interprets text. Encoder-only models excel here. defined in ch. 3 — open in glossary ) leaderboard. Its descendants each tweak the recipe:
- BERT pretrains on two objectives — masked language modeling masked language modeling A pretraining objective (used by BERT) that randomly masks tokens and trains the model to predict them from bidirectional context. defined in ch. 3 — open in glossary (predict hidden tokens) and next-sentence prediction.
- DistilBERT applies knowledge distillation knowledge distillation Training a small 'student' model to reproduce the behavior of a larger 'teacher', as used to build DistilBERT. defined in ch. 3 — open in glossary to keep 97% of BERT’s accuracy at 60% of the speed.
- RoBERTa simply trains BERT better — longer, more data, no NSP.
- XLM / XLM-R push into multilingual territory, XLM-R on 2.5 TB of text.
- ALBERT shrinks the parameter count with weight sharing; ELECTRA makes pretraining ~30× more efficient with a replaced-token discriminator; and DeBERTa’s disentangled attention was first to beat the human SuperGLUE baseline.
Encoder-only models still dominate NLU tasks like classification, NER, and extractive question answering — the subject of Chapter 7.
The decoder branch
The decoder-only decoder-only Models (the GPT family) that use causal attention to predict the next token; best for text generation. defined in ch. 3 — open in glossary line, led by OpenAI, is built for text generation ( NLG nlg Natural language generation — tasks like text generation and summarization where the model produces text. Decoder and encoder-decoder models excel here. defined in ch. 3 — open in glossary ) and has been driven relentlessly by scale:
- GPT paired the transformer decoder with transfer learning; GPT-2 scaled it up into a fluent generator.
- CTRL added control tokens for steerable style.
- GPT-3 scaled 100× to 175B parameters — guided by the same scaling power laws we’ll study in Chapter 11 — and revealed few-shot learning.
- GPT-Neo / GPT-J are EleutherAI’s open re-creations of GPT-3-scale models.
The encoder-decoder branch
Encoder–decoder encoder-decoder A model family with both an encoder and a decoder (e.g. T5, BART); maps one sequence to another, as in translation and summarization. defined in ch. 3 — open in glossary models keep both halves and shine on sequence-to-sequence tasks:
- T5 unifies every task as text-to-text; BART blends BERT-style and GPT-style pretraining via input corruption + reconstruction.
- M2M-100 translates directly among 100 languages; BigBird uses sparse attention to stretch the context window from 512 to 4,096 tokens — a preview of the efficiency techniques in Chapter 11.
| Key innovation | Payoff | |
|---|---|---|
| DistilBERT | Knowledge distillation | 97% of BERT, 40% smaller, 60% faster |
| RoBERTa | Train longer, drop NSP | Beats the original BERT |
| ALBERT | Parameter sharing + factorized embeddings | Bigger models, fewer parameters |
| ELECTRA | Replaced-token detection | ~30× more training-efficient |
| DeBERTa | Disentangled content/position attention | First to beat the human SuperGLUE baseline |
| GPT-3 | Scale to 175B + power laws | Few-shot learning |
| T5 | Everything as text-to-text | One architecture for all tasks |
| BigBird | Sparse (linear) attention | 512 → 4,096-token context |
Conclusion
We started at the heart of the Transformer — self-attention self-attention Attention where the queries, keys, and values all come from the same sequence; each output embedding is a context-dependent weighted average of the sequence. defined in ch. 3 — open in glossary — and built outward: query/key/value projections and scaled dot-product attention, multiple heads, the position-wise feed-forward layer, layer normalization and skip connections, and positional embeddings. We attached a classification head to the encoder body, looked at how the decoder adds masking and cross-attention, and surveyed the family of models that grew from these pieces.
You now have a working mental model of how a transformer works under the hood. Next, we put it to use: Chapter 7 builds a real question-answering system on top of an encoder body, and Chapter 11 asks where these models are headed.
Check yourself: meet the transformers
1.Almost every transformer model belongs to one of three families. Which are they?
2.What two objectives was the original BERT pretrained on?
3.How does DistilBERT stay fast and small while keeping most of BERT's accuracy?
4.GPT-3 scaled to 175B parameters. Besides fluent text, what capability did that scale reveal?
5.What is the core idea behind T5?
6.Why is BigBird notable, and how does it connect to later chapters?