Evaluating Generative Text Models: The Loss · Build an LLM From Scratch

Chapter 4 left us with a GPT that generates gibberish. Before training can fix that, we need a number that says how bad the gibberish is — a loss. This chapter (Stage 2 of the roadmap) follows a seven-step plan; this unit covers steps 1–3:

The chapter map (book p. 74): first make quality measurable (1–3), then train against that measure (4), then refine generation (5) and persist/load weights (6–7).

5.1.1 Text ↔ token utilities

Two small helpers wrap the ch2 tokenizer so the rest of the chapter reads cleanly:

</> text_to_token_ids and token_ids_to_text

import tiktoken
from previous_chapters import generate_text_simple

def text_to_token_ids(text, tokenizer):
  encoded = tokenizer.encode(text, allowed_special={'<|endoftext|>'})  #A
  encoded_tensor = torch.tensor(encoded).unsqueeze(0)  # add batch dimension
  return encoded_tensor

def token_ids_to_text(token_ids, tokenizer):
  flat = token_ids.squeeze(0)  # remove batch dimension
  return tokenizer.decode(flat.tolist())

start_context = "Every effort moves you"
tokenizer = tiktoken.get_encoding("gpt2")

token_ids = generate_text_simple(
  model=model,
  idx=text_to_token_ids(start_context, tokenizer),
  max_new_tokens=10,
  context_size=GPT_CONFIG_124M["context_length"]
)
print("Output text:\n", token_ids_to_text(token_ids, tokenizer))
# Output text:
#  Every effort moves you rentingetic wasn refres RexMeCHicular stren

#A allowed_special lets the <|endoftext|> marker from ch2 pass through the encoder instead of raising an error.

Still gibberish — but now we’ll measure exactly how gibberish.

5.1.2 Calculating the text generation loss

Two tiny training examples, in the ch2 input→target format (targets = inputs shifted one right):

</> Inputs and targets

inputs = torch.tensor([[16833, 3626, 6100],  # ["every effort moves",
                     [40,    1107, 588]])  #  "I really like"]

targets = torch.tensor([[3626, 6100, 345  ],  # [" effort moves you",
                      [1107, 588, 11311]])  #  " really like chocolate"]

inputs → targets (2×3 token IDs)

"every effort moves"	16833	3626	6100
"I really like"	40	1107	588
	pos 0	pos 1	pos 2

The model must predict, at every position, the corresponding target ID: 3626 after 16833, 6100 after 3626, 345 after 6100 — and likewise for row 2, ending in 11311 (' chocolate').

Run the forward pass and softmax the logits into probabilities — then ask the crucial question: how much probability did the model give the CORRECT next tokens?

</> From logits to the probabilities of the target tokens

with torch.no_grad():
  logits = model(inputs)
probas = torch.softmax(logits, dim=-1)
print(probas.shape)
# torch.Size([2, 3, 50257])  — (batch_size, num_tokens, vocab_size)

# what the untrained model WOULD generate (argmax): meaningless IDs
token_ids = torch.argmax(probas, dim=-1, keepdim=True)

# probability assigned to each TARGET token
text_idx = 0
target_probas_1 = probas[text_idx, [0, 1, 2], targets[text_idx]]  #A
print("Text 1:", target_probas_1)
# Text 1: tensor([2.3466e-05, 2.0531e-05, 1.1733e-05])

text_idx = 1
target_probas_2 = probas[text_idx, [0, 1, 2], targets[text_idx]]
print("Text 2:", target_probas_2)
# Text 2: tensor([4.2794e-05, 1.6248e-05, 1.1586e-05])

#A Fancy indexing: for sequence 0, at positions [0,1,2], pick the probability at the TARGET IDs [3626, 6100, 345]. Training aims to push these values toward 1 so the model consistently picks the target token.

Around 2 in 100,000 — the untrained model gives the right answers almost no probability mass. The loss pipeline turns these six numbers into one scalar. Step through it with the book’s exact values:

Logits — raw scores

The model outputs (2, 3, 50257) raw scores. Nothing is a probability yet.

1 / 6

The six-step loss pipeline (book p. 76). Steps 1–3 are already computed above; steps 4–6 produce the scalar that training will minimize.

Cross entropy, formally

$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \log\, p_\theta\!\left(t_i \mid \text{context}_i\right)$

symbol	meaning	here
$N$	number of scored positions	6 (2 sequences × 3 positions)
$p_\theta(t_i \mid \cdot)$	probability the model (parameters $\theta$ ) assigns to target token $t_i$	the six values of step 3
$\log$	natural log — products become sums, tiny values become workable	step 4
$\mathcal{L}$	the cross-entropy loss	10.7940

Worked example: $-\tfrac{1}{6}(-9.5042 - 10.3796 - 11.3677 - 11.4798 - 9.7764 - 12.2561) = 10.7940$ .

PyTorch’s cross_entropy fuses softmax + log + mean + negate — it takes raw logits and integer targets, flattened so batch and sequence merge:

</> The one-call version

# Logits: (batch_size, num_tokens, vocab_size); targets: (batch_size, num_tokens)
logits_flat = logits.flatten(0, 1)  #A
targets_flat = targets.flatten()
print("Flattened logits:", logits_flat.shape)
print("Flattened targets:", targets_flat.shape)
# Flattened logits: torch.Size([6, 50257])
# Flattened targets: torch.Size([6])

loss = torch.nn.functional.cross_entropy(logits_flat, targets_flat)  #B
print(loss)
# tensor(10.7940)

#A Every position is an independent 50,257-way classification, so batch and sequence dims merge: 6 predictions, 6 labels.
#B Softmax, target lookup, log, mean and negation — all inside one numerically-stable call. Same 10.7940 as the manual pipeline.

Perplexity — the interpretable twin

Perplexity =

e^{\mathcal{L}}

. Here

e^{10.7940} \approx 48{,}700

: the model is effectively guessing among ~48,700 tokens — barely better than uniform over the 50,257-token vocabulary. A trained model’s perplexity might be in the tens. Because it reads as an “effective vocabulary size”, perplexity is often reported alongside the raw loss.

5.1.3 Training and validation set losses

To train honestly we split the corpus — a large portion for training, a small held-out portion whose validation loss tells us whether the model generalizes rather than memorizes:

Recreation of the book’s p. 78 data figure: text → tokens → chunks (max_length 6 for illustration; 256 in the real loaders so the model sees longer texts) → shuffled batches, split into train/validation portions.

</> Loss over a batch, and over a whole data loader

def calc_loss_batch(input_batch, target_batch, model, device):
  input_batch = input_batch.to(device)  #A
  target_batch = target_batch.to(device)
  logits = model(input_batch)
  loss = torch.nn.functional.cross_entropy(
      logits.flatten(0, 1), target_batch.flatten())
  return loss

def calc_loss_loader(data_loader, model, device, num_batches=None):
  total_loss = 0.
  if len(data_loader) == 0:
      return float("nan")
  elif num_batches is None:
      num_batches = len(data_loader)
  else:
      num_batches = min(num_batches, len(data_loader))  #B
  for i, (input_batch, target_batch) in enumerate(data_loader):
      if i < num_batches:
          loss = calc_loss_batch(input_batch, target_batch, model, device)
          total_loss += loss.item()  #C
      else:
          break
  return total_loss / num_batches

with torch.no_grad():  #D
  train_loss = calc_loss_loader(train_loader, model, device)
  val_loss = calc_loss_loader(val_loader, model, device)

#A Move data to the same device as the model (CPU or GPU) before the forward pass.
#B num_batches caps the evaluation cost — §5.2 evaluates on just eval_iter=5 batches per checkpoint instead of the full loader.
#C .item() detaches the scalar from the graph; the running sum is plain Python.
#D Evaluation only — no gradients needed, so no_grad() saves memory and time.

Key idea — the loss is the training signal

One differentiable number now summarizes text quality: the negative average log-probability of the correct next tokens. Untrained: ≈ 10.79 (perplexity ≈ 48,700 — near-uniform guessing). §5.2 wires this number to backpropagation and an optimizer, and watches it fall.

📝 Check yourself: evaluating generative models

0 / 5

1.What quantity does the text-generation loss actually measure?
2.Why take the LOGARITHM of the target probabilities instead of averaging them directly?
3.Predict: torch.nn.functional.cross_entropy needs the logits flattened from (2, 3, 50257) to (6, 50257) and targets from (2, 3) to (6,). Why?
4.The untrained model's loss is ≈ 10.79, giving perplexity e^10.79 ≈ 48,700. What does that perplexity mean?
5.Why is part of the text held out as a VALIDATION set instead of training on everything?