§1.5–1.8Scale, the GPT Architecture, and the Roadmap

We now know an LLM is a pretrained-then-fine-tuned transformer. Two questions remain before we start building: what data makes pretraining work, and what exactly the GPT architecture is. Then §1.7 lays out the three-stage roadmap the rest of the book follows.

1.5 Utilizing large datasets

The “large” in LLM is as much about the corpus as the model. GPT-3 was pretrained on a blend of web crawls, books, and Wikipedia totalling hundreds of billions of tokens. The striking thing about the mix is that the proportion each source contributes to training is not the same as its raw size — only a fraction of each dataset is actually sampled, weighting quality over sheer volume. Click any row to see why.

Pretraining a model at this scale is expensive — training GPT-3 is estimated to have cost several million dollars in compute — which is exactly why, in Chapter 5, we will load OpenAI’s already-pretrained weights rather than pay to reproduce them, and only pretrain our small model on a tiny dataset to prove the mechanism works.

Emergent behavior

Something surprising happens as these models scale up. Trained only to predict the next word, large models begin to exhibit emergent behavior emergent behavior Abilities such as reasoning, arithmetic, or translation that a model exhibits only as its scale grows, without being explicitly trained for them. defined in ch. 1 — open in glossary — abilities like translation, arithmetic, and multi-step reasoning that were never explicitly part of the training objective. These capabilities are not attributable to any single parameter; they appear spontaneously as a product of scale and are largely absent in smaller models.

1.6 A closer look at the GPT architecture

The model we build is a GPT gpt A decoder-only transformer trained by next-word (autoregressive) prediction; strong at text generation. The model built in this book. defined in ch. 1 — open in glossary — a decoder-only transformer. It drops the encoder half of the original transformer entirely and keeps only the decoder, trained on the single self-supervised task of predicting the next word. Because each prediction is fed back in to help produce the next one, generation is autoregressive autoregressive Generating each token conditioned on all previously generated tokens — how GPT produces text. defined in ch. 1 — open in glossary : the model consumes its own output, one token at a time.

This deceptively simple objective is what enables zero-shot zero-shot Solving a task from the prompt alone, with no worked examples provided. defined in ch. 1 — open in glossary and few-shot few-shot Solving a task given a handful of examples in the prompt. defined in ch. 1 — open in glossary use: a large enough GPT can perform a brand-new task from the prompt alone (zero-shot) or from a handful of in-prompt examples (few-shot), with no weight updates at all — a direct consequence of the emergent behavior above.

1.7 Building an LLM — the three-stage roadmap

Here is the plan for the whole book. Keep this map in mind; every chapter slots into one of these three stages.

1.8 Summary

• An LLM is a deep neural network trained on massive text; “large” spans both its billions of parameters parameters The learnable weights of a neural network. GPT-3 has 175 billion of them. defined in ch. 1 — open in glossary and its huge corpus.
• LLMs rely on the transformer transformer The neural architecture behind modern LLMs — an encoder and/or decoder built around attention — introduced in "Attention Is All You Need" (2017). defined in ch. 1 — open in glossary and its self-attention self-attention A mechanism that lets each token weigh the relevance of every other token in the sequence. It is the core component of the transformer. defined in ch. 1 — open in glossary mechanism; deep learning removed the need for manual feature engineering.
• They are trained in two steps: pretraining pretraining The first training stage: self-supervised next-word prediction on large unlabeled text, producing a general-purpose foundation model. defined in ch. 1 — open in glossary on unlabeled text (self-supervised next-word prediction) yields a foundation model foundation model The general-purpose pretrained LLM, before any task-specific fine-tuning. defined in ch. 1 — open in glossary , which fine-tuning fine-tuning A later training stage on a smaller labeled dataset that specializes the pretrained model to a specific task. defined in ch. 1 — open in glossary then specializes.
• GPT is a decoder-only, autoregressive autoregressive Generating each token conditioned on all previously generated tokens — how GPT produces text. defined in ch. 1 — open in glossary transformer. At scale it shows emergent behavior emergent behavior Abilities such as reasoning, arithmetic, or translation that a model exhibits only as its scale grows, without being explicitly trained for them. defined in ch. 1 — open in glossary and can work zero- or few-shot.
• The book proceeds in three stages: build (ch2–4) → pretrain (ch5) → fine-tune (ch6–7). Next up: Chapter 2 turns raw text into the tokens and embeddings the model actually consumes.