Glossary

Every term this excerpt defines — 81 of them — with the chapter that introduced it. These are the same definitions behind the dotted-underline tooltips throughout the site.

81 shown

answer span ch. 7: The substring of the context marked as the answer, given by a start index and the answer text.
attention head ch. 3: One independent set of query/key/value projections; BERT uses 12 heads of dimension 768/12 = 64, each free to focus on a different relation.
attention scores ch. 3: The raw n×n dot products of queries with keys, before scaling and softmax turn them into attention weights.
attention weights ch. 3: Normalized coefficients w_ji (each row sums to 1) that say how much each token contributes to another token's updated embedding.
automatic speech recognition ch. 11: (ASR) Converting spoken audio into text; wav2vec 2.0 is a transformer-based approach using unlabeled pretraining.
bidirectional attention ch. 3: Attention in which a token's representation depends on both its left and right context.
bm25 ch. 7: 'Best Match 25', an improved TF-IDF that saturates term frequency and normalizes for document length; the default sparse retriever.
body ch. 3: The task-independent transformer trunk that produces hidden states, to which different task-specific heads are attached.
causal attention ch. 3: Autoregressive attention in which a token attends only to itself and earlier tokens, never to future ones.
causal mask ch. 3: A lower-triangular mask that sets attention scores to future tokens to −∞, so softmax gives them zero weight.
classification head ch. 3: A task-specific module (dropout + a linear layer on the [CLS] hidden state) attached to the transformer body to make class predictions.
closed-domain qa ch. 7: QA over a narrow topic or small document set, such as a single product category.
cls token ch. 3: A special token prepended to the input whose final hidden state is conventionally used to represent the whole sequence for classification.
contextualized embedding ch. 3: A token representation that incorporates information from its surrounding context, so homonyms like 'flies' get different vectors in different sentences.
contrastive learning ch. 11: Training that pulls matching pairs (e.g. an image and its caption) together in embedding space while pushing non-matching pairs apart, as in CLIP.
cross-entropy loss ch. 11: The standard training loss for language models; scaling laws chart it against compute, data, and model size.
decoder ch. 3: A stack of layers that generates an output sequence one token at a time, autoregressively, using the encoder's output and its own past outputs.
decoder-only ch. 3: Models (the GPT family) that use causal attention to predict the next token; best for text generation.
dense passage retrieval ch. 7: A dense retriever (DPR) using two BERT encoders — a bi-encoder — to embed the question and the passage and score them by similarity.
dense retriever ch. 7: A retriever that represents query and document as dense contextualized embeddings from a transformer (e.g. DPR).
document store ch. 7: A document-oriented database that holds the documents and metadata served to the retriever at query time.
domain adaptation ch. 7: Further fine-tuning a model on the target domain's data to close the gap left by training on a different domain.
efficient attention ch. 11: Techniques that reduce self-attention's O(n²) cost for long sequences, chiefly sparse and linearized attention.
encoder ch. 3: A stack of layers that turns input tokens into a sequence of contextualized embeddings (the hidden state / context).
encoder-decoder ch. 3: A model family with both an encoder and a decoder (e.g. T5, BART); maps one sequence to another, as in translation and summarization.
encoder-decoder architecture ch. 3: The original transformer form: an encoder builds a representation of the input and a decoder generates an output sequence from it. Used for tasks like translation.
encoder-decoder attention ch. 3: Cross-attention in the decoder where the queries come from the decoder while the keys and values come from the encoder's output.
encoder-only ch. 3: Models (the BERT family) that use bidirectional attention to build rich representations; best for understanding tasks like classification, NER, and QA.
exact match ch. 7: A strict QA metric (EM): 1 if the predicted answer exactly equals the ground truth after normalization, else 0.
extractive qa ch. 7: Question answering where the answer is a contiguous span of text located within a document.
f1-score ch. 7: The harmonic mean of precision and recall; for QA, computed over the token overlap between predicted and gold answers.
feed-forward layer ch. 3: A position-wise two-layer MLP (Linear → GELU → Linear) with hidden size about 4× the embedding size, applied to each token independently.
few-shot learning ch. 11: Performing a new task from just a few examples given in the prompt, without any weight updates; emerges in very large models like GPT-3.
gelu ch. 3: Gaussian Error Linear Unit, the smooth activation function used in the transformer's feed-forward sublayer.
generative qa ch. 7: Abstractive QA where a model generates a free-form answer rather than extracting a span from the text.
hidden state ch. 3: The sequence of embedding vectors the encoder produces for a sequence, also called the context.
key ch. 3: The learned projection of a token embedding that is 'matched against' by queries via a similarity (dot product).
knowledge distillation ch. 3: Training a small 'student' model to reproduce the behavior of a larger 'teacher', as used to build DistilBERT.
layer normalization ch. 3: A normalization that rescales each input to zero mean and unit variance across its features, stabilizing deep-network training.
learning rate warm-up ch. 3: Gradually increasing the learning rate from a small value to its maximum early in training, needed to stabilize post-LN transformers.
linearized attention ch. 11: Rewriting the attention similarity as a kernel φ(Q)ᵀφ(K) and reordering the sums so time and memory scale linearly in sequence length.
masked language modeling ch. 3: A pretraining objective (used by BERT) that randomly masks tokens and trains the model to predict them from bidirectional context.
masked self-attention ch. 3: Decoder self-attention with a causal mask so each position attends only to itself and earlier positions, preventing it from seeing future tokens.
mean average precision ch. 7: A retrieval metric (mAP) that rewards placing the correct answers higher up in the ranking.
multi-head attention ch. 3: Running several attention heads in parallel, each with its own Q/K/V projections, then concatenating and linearly projecting their outputs.
multimodal ch. 11: Describing models that combine more than one input modality, such as text with images or text with audio.
next sentence prediction ch. 3: A BERT pretraining task that predicts whether one text passage plausibly follows another; later found unnecessary by RoBERTa.
nlg ch. 3: Natural language generation — tasks like text generation and summarization where the model produces text. Decoder and encoder-decoder models excel here.
nlu ch. 3: Natural language understanding — tasks like classification, NER, and question answering where the model reads and interprets text. Encoder-only models excel here.
open-domain qa ch. 7: QA over a very large, general corpus, such as an entire product catalog or all of Wikipedia.
patch embedding ch. 11: The linear projection of a fixed-size image patch into a vector — ViT's analogue of a token embedding.
permutation equivariant ch. 3: A property of attention and feed-forward layers whereby permuting the input permutes the output identically — the reason positional information must be added.
positional embedding ch. 3: A position-dependent vector added to token embeddings so the otherwise order-blind attention mechanism can model token order.
post-layer normalization ch. 3: Placing layer normalization between the skip connections (the original Transformer arrangement); powerful but tricky to train, often needing warm-up.
power law ch. 11: A relationship y ∝ x^(−α) that appears as a straight line on log-log axes; language-model loss follows one in compute, data, and size.
pre-layer normalization ch. 3: Placing layer normalization inside the span of the skip connections; the most common arrangement, much more stable to train.
query ch. 3: The learned projection of a token embedding that does the 'asking' — matched against every key to decide how much to attend.
question answering ch. 7: Finding the answer to a natural-language question, often by searching and reading a corpus of documents.
reader ch. 7: The QA component (a reading-comprehension model) that extracts an answer span from retrieved documents.
recall ch. 7: In retrieval, the fraction of all relevant documents that appear in the retriever's top-k results.
retrieval-augmented generation ch. 7: Generative QA (RAG) that feeds DPR-retrieved documents to a seq2seq generator (T5/BART), trainable end-to-end.
retriever ch. 7: The QA component that fetches the documents most relevant to a query.
retriever-reader architecture ch. 7: The two-stage QA design: a retriever fetches relevant documents, then a reader extracts the answer from them.
sample efficiency ch. 11: Reaching a given performance with fewer training steps or examples; larger models tend to be more sample-efficient.
scaled dot-product attention ch. 3: The standard self-attention recipe: score queries against keys with a dot product, scale by 1/√d_k, softmax to get weights, then take the weighted sum of values.
scaling laws ch. 11: Empirical power-law relationships between a language model's loss and its compute budget, dataset size, and parameter count.
self-attention ch. 3: Attention where the queries, keys, and values all come from the same sequence; each output embedding is a context-dependent weighted average of the sequence.
skip connection ch. 3: A residual connection that adds a sublayer's input to its output, letting gradients flow and easing training of deep stacks.
sliding window ch. 7: Splitting a long context into overlapping token windows (via stride) so no window exceeds the model's maximum length.
span classification ch. 7: Framing extractive QA as predicting the start-token and end-token of the answer span.
sparse attention ch. 11: Computing attention over only a chosen subset of query-key pairs (global, band, dilated, random, or block-local patterns) to cut cost.
sparse retriever ch. 7: A retriever that represents query and document as sparse word-frequency vectors (e.g. TF-IDF, BM25).
squad ch. 7: The Stanford Question Answering Dataset, the canonical extractive-QA benchmark; v2 adds unanswerable questions.
tf-idf ch. 7: Term frequency–inverse document frequency: a classic scheme weighting words by how often they occur in a document versus the corpus.
the bitter lesson ch. 11: Rich Sutton's observation that general methods leveraging ever-more computation beat hand-built human-knowledge approaches in the long run.
token embedding ch. 3: The fixed vector a token id maps to via a lookup table (768-dimensional in BERT), before any context is mixed in.
token_type_ids ch. 7: A per-token tensor marking which input tokens belong to the question (0) versus the context (1) in a QA pair.
transformer ch. 3: A neural architecture built on self-attention (no recurrence or convolution) that maps a sequence of token embeddings to a sequence of contextualized embeddings.
value ch. 3: The learned projection of a token embedding that carries the content mixed into the output according to the attention weights.
vision transformer ch. 11: (ViT) A BERT-style transformer for images that splits the image into patches, linearly embeds each as a token, and runs a standard encoder.
zero-shot learning ch. 11: Performing a task with no task-specific training examples — e.g. CLIP classifying images into classes defined only by text.