Glossary

Every term the book defines — 114 of them — with the chapter that introduced it. These are the same definitions behind the dotted-underline tooltips throughout the site.

114 shown

absolute positional embedding ch. 2: One learned vector per position index — the scheme GPT uses; shape (context_length, emb_dim).
adamw ch. 5: An Adam-optimizer variant with improved (decoupled) weight decay — the standard optimizer for training LLMs.
attention mechanism ch. 3: Lets a model directly access and weight all input positions when building each representation, instead of relying on one compressed hidden state.
attention score ch. 3: ω — the raw query·key dot product between two tokens, before normalization.
attention weight ch. 3: α — the softmax-normalized attention scores; each query's row of weights sums to 1.
autograd ch. A: PyTorch's automatic differentiation engine — computes gradients of tensor operations automatically.
autoregressive ch. 1: Generating each token conditioned on all previously generated tokens — how GPT produces text.
backpropagation ch. A: The chain rule applied over the computation graph to obtain the loss's gradient with respect to every parameter.
batch size ch. 2: How many sequences are processed together in one training step; small batches use less memory but give noisier updates.
bert ch. 1: An encoder-only transformer trained by masked-word prediction; strong at classification and language understanding.
bpe ch. 2: Byte pair encoding — a sub-word tokenizer that merges frequent adjacent symbol pairs; GPT-2 uses a 50,257-token BPE vocabulary.
byte pair encoding ch. 2: A sub-word tokenizer (BPE) that iteratively merges the most frequent adjacent symbol pair. It can represent any word yet keeps common words whole; GPT-2's BPE vocabulary has 50,257 tokens.
causal attention ch. 3: Masked attention: self-attention restricted so each token only attends to itself and previous positions, never future ones.
causal mask ch. 3: The upper-triangular matrix marking future positions; those attention scores are set to −∞ before softmax so their weights become 0.
classification accuracy ch. 6: The fraction of correct predictions over a dataset — the evaluation metric, while differentiable cross-entropy remains the training loss.
classification fine-tuning ch. 1: Fine-tuning on (text, class-label) pairs for a categorization task such as spam detection (see ch6).
classification head ch. 6: The replacement output layer (Linear 768 → num_classes) that maps the last token's representation to class logits.
collate function ch. 7: The DataLoader hook that assembles individual dataset items into a batch; customized in ch7 to pad dynamically and build masked targets.
computation graph ch. A: The directed graph of operations PyTorch records during the forward pass; gradients flow back through it during backpropagation.
context length ch. 2: The maximum number of tokens the model can accept as input (a.k.a. max_length). GPT-2 supports 1024.
context vector ch. 3: The weighted sum over all tokens' (value) vectors — an enriched representation of one token that incorporates information from the whole sequence.
cosine decay ch. D: Decreasing the learning rate from its peak toward a minimum along a half-cosine curve after warmup, slowing learning as the model converges.
cross-entropy loss ch. 5: The negative average log-probability the model assigns to the correct next tokens — the objective minimized during pretraining.
decoder ch. 1: The transformer half that generates output tokens one at a time. GPT is decoder-only; this book builds a decoder-only model.
deep learning ch. 1: Machine learning with multi-layer neural networks that learn their own representations from raw data, without manual feature engineering.
distributed training ch. A: Dividing model training across multiple GPUs or machines (e.g. PyTorch's DistributedDataParallel).
dot product ch. 3: Element-wise multiplication of two vectors followed by summing — a measure of how aligned (similar) the vectors are.
dropout ch. 3: Randomly zeroing units (here: attention weights) during training to prevent overfitting; the survivors are rescaled by 1/(1−p). Disabled at inference.
dynamic padding ch. 7: Padding each batch to its own longest sequence instead of one global length — different batches then have different widths.
embedding ch. 2: A mapping from discrete data (tokens) to points in a continuous vector space, arranged so that related items sit near each other.
embedding dimension ch. 2: The length of each embedding vector (emb_dim / output_dim). GPT-2 small uses 768.
emergent behavior ch. 1: Abilities such as reasoning, arithmetic, or translation that a model exhibits only as its scale grows, without being explicitly trained for them.
encoder ch. 1: The transformer half that maps an input sequence into context-carrying embedding vectors. BERT is encoder-only.
epoch ch. 5: One complete pass over the training data loader.
feed forward network ch. 4: The per-position Linear → GELU → Linear module of a transformer block, expanding 4× then contracting (768→3072→768 in GPT-2 small).
few-shot ch. 1: Solving a task given a handful of examples in the prompt.
fine-tuning ch. 1: A later training stage on a smaller labeled dataset that specializes the pretrained model to a specific task.
foundation model ch. 1: The general-purpose pretrained LLM, before any task-specific fine-tuning.
gelu ch. 4: Gaussian Error Linear Unit — the smooth activation ≈ 0.5x(1+tanh(√(2/π)(x+0.044715x³))) used in GPT's feed-forward layers.
generative ai ch. 1: AI that creates new content such as text, images, or code. LLMs are the text-generation instance of generative AI.
gpt ch. 1: A decoder-only transformer trained by next-word (autoregressive) prediction; strong at text generation. The model built in this book.
gradient ch. A: The vector of all partial derivatives of a multivariate function — the direction gradient descent follows to reduce the loss.
gradient clipping ch. D: Rescaling gradients so their L2 norm never exceeds a threshold (max_norm), preventing exploding updates.
greedy decoding ch. 4: Generating by always picking the highest-probability next token (argmax); applying softmax first is optional since it doesn't change which token wins.
head dimension ch. 3: head_dim = d_out / num_heads — the slice of the projection each attention head works with in the weight-split implementation.
ignore_index ch. 7: The target value (−100, PyTorch's cross_entropy default) that the loss function skips; used to exclude padding tokens from training.
input embedding ch. 2: The element-wise sum of a token embedding and its positional embedding — the actual numeric input to the model.
input-target pair ch. 2: A training example whose target is the input sequence shifted one token to the right — the setup for next-word prediction.
instruction fine-tuning ch. 1: Fine-tuning on (instruction, response) pairs so the model learns to follow natural-language instructions (see ch7).
key ch. 3: The projection x·W_k each token exposes to be matched against queries when computing attention scores.
l2 norm ch. D: The Euclidean length of a vector or matrix: the square root of the sum of squared entries.
large language model ch. 1: A deep neural network trained on massive amounts of text to understand and generate human-like language; "large" refers to both its billions of parameters and its huge training corpus.
layer freezing ch. 6: Setting requires_grad=False so a layer's weights stay fixed during fine-tuning; only the unfrozen layers learn.
layer normalization ch. 4: Normalizing each token's activations to mean 0 and unit variance across the feature dimension, then applying a learnable scale and shift — stabilizes deep-network training.
learning rate warmup ch. D: Linearly increasing the learning rate from a tiny initial value to the peak over the first steps of training, avoiding destabilizing early updates.
llm ch. 1: Large language model — a deep neural network trained on massive text to understand and generate human-like language; "large" = billions of parameters + a huge training corpus.
llm-as-a-judge ch. 7: Evaluating a model's free-form responses by asking another LLM to score them against reference outputs (e.g. on a 0–100 scale).
logits ch. 4: The raw, pre-softmax scores over the vocabulary that the model outputs at each position.
lora ch. E: Low-rank adaptation: approximating a layer's weight update as a product of two small matrices (ΔW ≈ AB), trained while the original weights stay frozen.
lora alpha ch. E: The scaling factor applied to the low-rank path's output, regulating its influence on the adapted layer.
lora rank ch. E: The inner dimension r of the A and B matrices — controls how many parameters LoRA adds and how much adaptation capacity it has.
model.eval() ch. 4: Switches a PyTorch module to inference mode, disabling train-only behavior such as dropout.
multi-head attention ch. 3: Several causal-attention heads run in parallel, each with its own Q/K/V projections; their outputs are concatenated and mixed by an output projection.
multilayer perceptron ch. A: A stack of fully-connected (Linear) layers with nonlinear activations between them.
nn.embedding ch. 2: The PyTorch layer implementing a trainable lookup table; used for both token and positional embeddings.
output projection ch. 3: The final linear layer (out_proj) of multi-head attention that mixes the concatenated head outputs.
overfitting ch. 5: Memorizing the training data instead of generalizing: training loss keeps dropping while validation loss doesn't budge.
padding ch. 6: Appending a pad token (here 50256, <|endoftext|>) to shorter sequences so every sequence in a batch has the same length.
parameter-efficient fine-tuning ch. E: Adapting a large model by training only a small fraction of (or small addition to) its parameters.
parameters ch. 1: The learnable weights of a neural network. GPT-3 has 175 billion of them.
partial derivative ch. A: A function's rate of change with respect to one of its variables.
perplexity ch. 5: exp(cross-entropy loss); an interpretable uncertainty measure — roughly the effective number of tokens the model is choosing among.
positional embedding ch. 2: A vector added to a token embedding to encode the token's position in the sequence.
preference fine-tuning ch. 7: An optional stage after instruction fine-tuning that further aligns a model with specific user preferences.
pretraining ch. 1: The first training stage: self-supervised next-word prediction on large unlabeled text, producing a general-purpose foundation model.
probabilistic sampling ch. 5: Drawing the next token from the probability distribution (torch.multinomial) instead of always taking the argmax.
prompt style ch. 7: The template that flattens (instruction, input, response) into one training text — e.g. Alpaca's ### sections or Phi-3's <|user|>/<|assistant|> tags.
query ch. 3: The projection x·W_q of the token currently probing the sequence — it is matched against every key to decide where to attend.
register_buffer ch. 3: PyTorch mechanism for storing a non-trainable tensor (like the causal mask) on a module so it moves with the model across devices.
relative positional embedding ch. 2: Position encoding based on the distance between tokens rather than their absolute index; generalizes better to unseen lengths.
relu ch. A: The activation max(0, x) — the classic hard-hinge ancestor of GELU.
scaled dot-product attention ch. 3: softmax(QKᵀ/√d_k)·V — self-attention with trainable projections; dividing by √d_k keeps softmax out of its vanishing-gradient regime at large dimensions.
self-attention ch. 1: A mechanism that lets each token weigh the relevance of every other token in the sequence. It is the core component of the transformer.
self-supervised learning ch. 1: Training where the labels are derived from the data itself (e.g. predicting the next word), requiring no human annotation.
sgd ch. A: Stochastic gradient descent — the plain optimizer (contrast AdamW, which adds adaptive moments and decoupled weight decay).
shortcut connection ch. 4: Adding a layer's input to its output (x + f(x)); a.k.a. skip or residual connection. Keeps gradients flowing in deep stacks.
sliding window ch. 2: Moving a fixed-size window across the token stream (in steps of 'stride') to generate many overlapping input-target training samples.
softmax ch. 3: Exponentiate-and-normalize: turns a vector of scores into positive probabilities that sum to 1.
special token ch. 2: A reserved token marking structure or unknown words, e.g. <|endoftext|>, <|unk|>, [BOS], [EOS], [PAD].
state_dict ch. 5: PyTorch's name→tensor dictionary of a module's or optimizer's parameters, used with torch.save/load for checkpointing.
stride ch. 2: The number of tokens the sliding window advances between consecutive training samples.
sub-word tokenization ch. 2: Tokenizing into pieces smaller than whole words, so even unseen words can be represented as a sequence of known sub-words.
temperature scaling ch. 5: Dividing the logits by a temperature T before softmax: T>1 flattens the distribution (more diverse output), T<1 sharpens it (more deterministic).
tensor ch. A: PyTorch's n-dimensional array (NumPy-like), with GPU acceleration and autograd integration.
test set ch. 6: A third data split evaluated only at the very end, giving an unbiased estimate of real-world performance.
tiktoken ch. 2: OpenAI's fast BPE tokenizer library (with a Rust core) used in the book instead of reimplementing BPE.
token ch. 1: A unit of text (a word or sub-word) that the model reads and produces. Formalized in ch2.
token embedding ch. 2: The vector a learned nn.Embedding lookup assigns to a token ID. The embedding matrix has shape (vocab_size, emb_dim).
token id ch. 2: The integer that a vocabulary assigns to a token.
tokenization ch. 2: Splitting raw text into tokens (words, sub-words, or characters) as the first step of turning text into numbers.
top-k sampling ch. 5: Restricting sampling to the k highest-scoring tokens by masking all others to −∞ before the softmax.
top-p sampling ch. 5: Nucleus sampling: restricting sampling to the smallest set of tokens whose cumulative probability reaches p.
torch.no_grad() ch. A: Context manager that disables gradient tracking for inference, saving memory and compute.
transformer ch. 1: The neural architecture behind modern LLMs — an encoder and/or decoder built around attention — introduced in "Attention Is All You Need" (2017).
transformer block ch. 4: The repeated unit of a GPT: LayerNorm → multi-head attention → dropout → residual add, then LayerNorm → feed-forward → dropout → residual add. Preserves the (batch, tokens, emb_dim) shape.
truncation ch. 6: Cutting sequences down to a maximum length before padding.
validation loss ch. 5: The loss on held-out text never trained on. Training loss falling while validation loss stalls is the signature of overfitting.
value ch. 3: The projection x·W_v carrying the content that is actually retrieved and mixed into the context vector, weighted by attention.
vanishing gradient ch. 4: Gradients shrinking toward zero as they propagate backward through many layers, starving early layers of learning signal.
vocabulary ch. 2: The set of all unique tokens, each mapped to a unique integer ID.
weight decay ch. 5: Penalizing large weights during optimization to limit model complexity and prevent overfitting.
weight tying ch. 4: Reusing the token-embedding matrix as the output head's weights (as the original GPT-2 did), saving ~38M parameters at the 124M scale.
word embedding ch. 2: An embedding of individual words or tokens — the most common form of text embedding.
zero-shot ch. 1: Solving a task from the prompt alone, with no worked examples provided.