Build a Large Language Model (From Scratch)

What a large language model is, why the transformer changed everything, and the two-stage recipe — pretraining then fine-tuning — that the rest of the book builds end to end.

• 1 Understanding Large Language Models pp. 3–6

Turning raw text into tensors a network can learn from, then the heart of the transformer — tokenization, byte-pair encoding, embeddings, and self-attention built up from dot products to multi-head causal attention.

• 2 Working with Text Data pp. 7–20
• 3 Coding Attention Mechanisms pp. 21–47

Assembling the full GPT: layer norm, GELU feed-forward blocks, shortcut connections and the transformer block, then pretraining it on unlabeled text with a cross-entropy loss, sampling strategies, and loading OpenAI weights.

• 4 Implementing a GPT Model from Scratch pp. 48–71
• 5 Pretraining on Unlabeled Data pp. 72–91

Adapting the pretrained model to do useful work — fine-tuning it as a spam classifier and as an instruction-follower — plus PyTorch fundamentals, training-loop refinements, and LoRA.

• 6 Fine-Tuning for Classification pp. 90–107
• 7 Fine-Tuning to Follow Instructions pp. 108–128
• A Introduction to PyTorch 📎 pp. 128–138
• D Bells and Whistles to the Training Loop 📎 pp. 141–146
• E Parameter-Efficient Fine-Tuning with LoRA 📎 pp. 147–157