§2.1–2.4Embeddings, Tokens, and Token IDs

A model multiplies numbers, not letters. This section builds the first two links of the chain that turns a raw string into something a GPT can consume: text → tokens → token IDs. (The next sections add sub-word tokenization and, finally, the embedding vectors themselves.)

2.1 Understanding word embeddings

An embedding embedding A mapping from discrete data (tokens) to points in a continuous vector space, arranged so that related items sit near each other. defined in ch. 2 — open in glossary maps discrete things — here, words — to points in a continuous vector space, arranged so that related items land near each other. A word embedding word embedding An embedding of individual words or tokens — the most common form of text embedding. defined in ch. 2 — open in glossary is the per-word case (you can also embed sentences or whole documents). Embeddings can have anywhere from one to thousands of dimensions; below we squash them to 2-D just to look at them.

2.2 Tokenizing text

Tokenization tokenization Splitting raw text into tokens (words, sub-words, or characters) as the first step of turning text into numbers. defined in ch. 2 — open in glossary is the first concrete step: splitting the raw string into tokens. A simple word/punctuation tokenizer is just a regular expression that splits on whitespace and keeps punctuation as its own tokens.

import re

text = "Hello, world. This, is a test."
result = re.split(r'([,.:;?_!"()\']|--|\s)', text)   #A
result = [item.strip() for item in result if item.strip()]   #B
print(result)
# ['Hello', ',', 'world', '.', 'This', ',', 'is', 'a', 'test', '.']

2.3 Converting tokens into token IDs

Tokens are still strings. To get numbers, collect every unique token into a vocabulary vocabulary The set of all unique tokens, each mapped to a unique integer ID. defined in ch. 2 — open in glossary that maps each token to a unique integer — its token ID token id The integer that a vocabulary assigns to a token. defined in ch. 2 — open in glossary . The vocabulary is usually sorted, so IDs are assigned alphabetically.

The book wraps this into a SimpleTokenizerV1 class with two methods: encode (text → IDs) and decode (IDs → text, via the inverse vocabulary).

class SimpleTokenizerV1:
  def __init__(self, vocab):
      self.str_to_int = vocab                              #A
      self.int_to_str = {i: s for s, i in vocab.items()}   #B

  def encode(self, text):                                  #C
      preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', text)
      preprocessed = [item.strip() for item in preprocessed if item.strip()]
      ids = [self.str_to_int[s] for s in preprocessed]
      return ids

  def decode(self, ids):                                   #D
      text = " ".join([self.int_to_str[i] for i in ids])
      text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)      #E
      return text

Encoding a sentence gives a plain list of integers — for example "the quick brown" might become:

2.4 Adding special context tokens

SimpleTokenizerV1 has a fatal flaw: feed it a word that isn’t in the vocabulary and it throws a KeyError. Real text also needs markers for structure — where one document ends and another begins. Both are solved with special tokens special token A reserved token marking structure or unknown words, e.g. <|endoftext|>, <|unk|>, [BOS], [EOS], [PAD]. defined in ch. 2 — open in glossary .

SimpleTokenizerV2 adds <|unk|> and <|endoftext|> to the vocabulary and, in encode, replaces any out-of-vocabulary token with <|unk|>:

all_tokens = sorted(set(preprocessed))
all_tokens.extend(["<|endoftext|>", "<|unk|>"])                 #A
vocab = {token: integer for integer, token in enumerate(all_tokens)}

class SimpleTokenizerV2:
  def __init__(self, vocab):
      self.str_to_int = vocab
      self.int_to_str = {i: s for s, i in vocab.items()}

  def encode(self, text):
      preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', text)
      preprocessed = [item.strip() for item in preprocessed if item.strip()]
      preprocessed = [
          item if item in self.str_to_int else "<|unk|>"        #B
          for item in preprocessed
      ]
      ids = [self.str_to_int[s] for s in preprocessed]
      return ids

  def decode(self, ids):
      text = " ".join([self.int_to_str[i] for i in ids])
      text = re.sub(r'\s+([,.:;?!"()\'])', r'\1', text)
      return text

Joining two texts with " <|endoftext|> ".join((text1, text2)) and encoding them shows both special tokens in action — unknown words map to the <|unk|> ID, and the <|endoftext|> ID marks the seam between the two texts.