§7.9–7.10Conclusions & Chapter Summary

The judge has scored the responses; the assistant works. This short closer marks the end of the book’s main arc — data → attention → architecture → pretraining → two kinds of fine-tuning — and points at the one step it leaves open.

7.9 Conclusions — what comes after instructions

The book covered the most essential steps, but one optional stage can follow instruction fine-tuning: preference fine-tuning preference fine-tuning An optional stage after instruction fine-tuning that further aligns a model with specific user preferences. defined in ch. 7 — open in glossary — particularly useful for customizing a model to better align with specific user preferences. An instruction-tuned model answers correctly; a preference-tuned model answers correctly in the way its users prefer — tone, style, and choices between equally-valid responses. (Implementing it is beyond the book’s scope — a deliberate open door.)

7.10 Summary

• Instruction fine-tuning adapts a pretrained LLM to follow human instructions and generate desired responses.
• Data preparation: download an instruction-response dataset, format the entries (Alpaca template), split into train/validation/test.
• Batches come from a custom collate function that pads dynamically, builds shifted target IDs, and masks padding with −100.
• The starting point is a pretrained GPT-2 medium (355M) — small models lack the capacity for high-quality instruction-following.
• Training uses a loop similar to pretraining — same loss, same optimizer, new data.
• Evaluation: extract test-set responses and score them with another LLM (an 8B-parameter Llama via Ollama), averaging for a quantitative result.