§2.5–2.6Byte Pair Encoding & Sliding-Window Sampling

The word-level tokenizer of §2.4 has two weaknesses: it needs an <|unk|> escape hatch for unseen words, and its vocabulary must list every word form separately. Real GPT models use byte pair encoding instead, and then slice the token stream into training examples with a sliding window.

2.5 Byte pair encoding

Byte pair encoding byte pair encoding A sub-word tokenizer (BPE) that iteratively merges the most frequent adjacent symbol pair. It can represent any word yet keeps common words whole; GPT-2's BPE vocabulary has 50,257 tokens. defined in ch. 2 — open in glossary (BPE) is sub-word tokenization sub-word tokenization Tokenizing into pieces smaller than whole words, so even unseen words can be represented as a sequence of known sub-words. defined in ch. 2 — open in glossary : instead of whole words, its vocabulary is made of pieces — from single characters up to common whole words. Any string can be spelled out of these pieces, so BPE never meets a word it can’t encode and needs no <|unk|> token.

How the vocabulary is built (training). BPE starts with a base vocabulary of individual bytes/characters, then repeats one step: find the most frequent adjacent pair of symbols in the corpus and merge it into a single new token. Frequent letter groups (like es, then est) and eventually whole common words get their own tokens, while rare words stay split into small pieces. GPT-2’s BPE runs this until the vocabulary reaches 50,257 tokens.

At inference, BPE simply applies the learned merges. Feed it an invented word and it falls back to sub-word pieces — here the book’s "Akwirw ier" example:

In practice the book doesn’t reimplement BPE — it uses OpenAI’s fast tiktoken tiktoken OpenAI's fast BPE tokenizer library (with a Rust core) used in the book instead of reimplementing BPE. defined in ch. 2 — open in glossary library (its core is written in Rust):

import tiktoken

tokenizer = tiktoken.get_encoding("gpt2")                    #A
text = "Hello, do you like tea? <|endoftext|> In the sunlit terraces."
integers = tokenizer.encode(text, allowed_special={"<|endoftext|>"})   #B
print(integers)
# [15496, 11, 466, 345, 588, 8887, 30, 220, 50256, 554, 262, ...]

strings = tokenizer.decode(integers)                         #C
print(strings)   # round-trips back to the original text

2.6 Data sampling with a sliding window

We can now turn text into a long stream of token IDs. To train next-word prediction we need (input, target) examples, where — since the model predicts the next token — the target is the input shifted right by one input-target pair A training example whose target is the input sequence shifted one token to the right — the setup for next-word prediction. defined in ch. 2 — open in glossary . A sliding window sliding window Moving a fixed-size window across the token stream (in steps of 'stride') to generate many overlapping input-target training samples. defined in ch. 2 — open in glossary of width max_length slides across the stream in steps of stride stride The number of tokens the sliding window advances between consecutive training samples. defined in ch. 2 — open in glossary , cutting out one pair per position.

Play with the two knobs below. Watch how max_length (the model’s context length context length The maximum number of tokens the model can accept as input (a.k.a. max_length). GPT-2 supports 1024. defined in ch. 2 — open in glossary ) sets how much the model sees, and how stride controls the overlap between consecutive training windows:

Within a single window, the pair itself is really n next-word predictions stacked up — the model may use “LLMs” to predict “learn”, “LLMs learn” to predict “to”, and so on, but it can never peek past its target:

The book packages this as a PyTorch Dataset that precomputes every window:

import torch
from torch.utils.data import Dataset, DataLoader

class GPTDatasetV1(Dataset):
  def __init__(self, txt, tokenizer, max_length, stride):
      self.input_ids = []
      self.target_ids = []
      token_ids = tokenizer.encode(txt)                          #A

      for i in range(0, len(token_ids) - max_length, stride):    #B
          input_chunk  = token_ids[i : i + max_length]
          target_chunk = token_ids[i + 1 : i + max_length + 1]   #C
          self.input_ids.append(torch.tensor(input_chunk))
          self.target_ids.append(torch.tensor(target_chunk))

  def __len__(self):
      return len(self.input_ids)                                 #D

  def __getitem__(self, idx):
      return self.input_ids[idx], self.target_ids[idx]

A thin wrapper builds the DataLoader, which handles batching (grouping batch_size pairs), shuffling, and iteration:

def create_dataloader_v1(txt, batch_size=4, max_length=256,
                       stride=128, shuffle=True, drop_last=True,
                       num_workers=0):
  tokenizer = tiktoken.get_encoding("gpt2")
  dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
  dataloader = DataLoader(
      dataset, batch_size=batch_size, shuffle=shuffle,
      drop_last=drop_last, num_workers=num_workers)              #A
  return dataloader

A batch batch size How many sequences are processed together in one training step; small batches use less memory but give noisier updates. defined in ch. 2 — open in glossary stacks several windows so the GPU processes them in parallel — for batch_size=8, max_length=4, stride=4 the inputs tensor has shape (8, 4) and the targets tensor is the same, shifted: