§11.1Scaling Transformers: Scaling Laws & Challenges

This final chapter looks at where transformers are going, along three axes: scaling them up, making attention efficient, and pushing them into new modalities. We start with scale.

The bitter lesson

The bitter lesson the bitter lesson Rich Sutton's observation that general methods leveraging ever-more computation beat hand-built human-knowledge approaches in the long run. defined in ch. 11 — open in glossary is playing out with transformers: while early BERT/GPT descendants tweaked architectures and objectives, the best 2021 models (GPT-3) are essentially scaled-up originals. Model size has grown by four-plus orders of magnitude since the 2017 Transformer:

This is driven by evidence that larger models do better downstream, and that zero-shot zero-shot learning Performing a task with no task-specific training examples — e.g. CLIP classifying images into classes defined only by text. defined in ch. 11 — open in glossary and few-shot few-shot learning Performing a new task from just a few examples given in the prompt, without any weight updates; emerges in very large models like GPT-3. defined in ch. 11 — open in glossary abilities emerge in the 10–100B-parameter range. But parameters aren’t the only lever — compute and data must scale too. Since a GPT-3-sized run costs ~$4.6M, it pays to predict performance in advance. Remarkably, you can.

Scaling laws

Scaling laws scaling laws Empirical power-law relationships between a language model's loss and its compute budget, dataset size, and parameter count. defined in ch. 11 — open in glossary quantify “bigger is better” by charting the cross-entropy loss cross-entropy loss The standard training loss for language models; scaling laws chart it against compute, data, and model size. defined in ch. 11 — open in glossary L against three factors: compute budget C, dataset size D, and model size N. Each follows a power law power law A relationship y ∝ x^(−α) that appears as a straight line on log-log axes; language-model loss follows one in compute, data, and size. defined in ch. 11 — open in glossary — a straight line on log-log axes:

Explore it: pick an axis and drag the exponent α. The book’s fitted values are α ≈ 0.050 (compute), 0.095 (data), 0.076 (parameters):

Three takeaways:

• Scale beats tweaking. Rather than tuning layers or heads on a fixed dataset, increasing N, C, and D in tandem is the more productive path.
• Smooth power laws. L is a clean power law over several orders of magnitude, so the early part of a loss curve extrapolates to predict the loss of a much longer (or larger) run — before paying for it.
• Sample efficiency sample efficiency Reaching a given performance with fewer training steps or examples; larger models tend to be more sample-efficient. defined in ch. 11 — open in glossary . Larger models reach a given loss in fewer training steps.

The challenges of scaling

“Just add more layers” is easy to say and hard to do. Scaling a language model runs into very practical walls:

Because these barriers concentrate power in a few well-resourced labs, community efforts like BigScience (an open research workshop) and EleutherAI (a decentralized collective behind GPT-Neo/GPT-J) aim to build and probe large models in the open.

Scale is one frontier; the other is making the attention mechanism itself cheaper — the subject of efficient self-attention.