§2.7–2.9Token & Positional Embeddings

We have token IDs and a way to batch them. The last step of Chapter 2 turns those integers into the continuous vectors the model actually multiplies — first the token embeddings, then the positional embeddings that tell the model where each token sits.

2.7 Creating token embeddings

A token embedding token embedding The vector a learned nn.Embedding lookup assigns to a token ID. The embedding matrix has shape (vocab_size, emb_dim). defined in ch. 2 — open in glossary is produced by an nn.Embedding nn.Embedding The PyTorch layer implementing a trainable lookup table; used for both token and positional embeddings. defined in ch. 2 — open in glossary layer, which is nothing more than a learned lookup table: a weight matrix of shape (vocab_size, emb_dim) where row i is the embedding of token ID i. Looking up an embedding is just selecting a row — equivalent to multiplying a one-hot vector by the matrix, but far cheaper.

Take a toy layer with vocab_size = 6 and embedding dimension embedding dimension The length of each embedding vector (emb_dim / output_dim). GPT-2 small uses 768. defined in ch. 2 — open in glossary emb_dim = 3. Its weight matrix starts random (here, after torch.manual_seed(123)):

Applying it to input_ids = [2, 3, 5, 1] stacks those four rows into a (4 × 3) tensor:

import torch

input_ids = torch.tensor([2, 3, 5, 1])
vocab_size = 6
output_dim = 3

torch.manual_seed(123)
embedding_layer = torch.nn.Embedding(vocab_size, output_dim)   #A

embedding_layer(torch.tensor([3]))     #B
# tensor([[-0.4015,  0.9666, -1.1481]], grad_fn=<EmbeddingBackward0>)

embedding_layer(input_ids)             #C
# tensor([[ 1.2753, -0.2010, -0.1606],
#         [-0.4015,  0.9666, -1.1481],
#         [-2.8400, -0.7849, -1.4096],
#         [ 0.9178,  1.5810,  1.3010]], grad_fn=<EmbeddingBackward0>)

2.8 Encoding word positions

The lookup above has a subtle gap: it is position-independent. Token ID 5 gets the vector [-2.84, -0.78, -1.41] whether it is the first or the fiftieth token. That is fine for reproducibility, but self-attention (Chapter 3) is itself order-agnostic — it has no built-in sense of sequence — so we must inject position information explicitly. There are two families:

The recipe is delightfully simple: build a second embedding layer indexed by position instead of by token, then add the two. A positional-embedding layer of shape (context_length, emb_dim) produces one vector per slot; the input embedding input embedding The element-wise sum of a token embedding and its positional embedding — the actual numeric input to the model. defined in ch. 2 — open in glossary is the element-wise sum.

vocab_size = 50257
output_dim = 256
max_length = 4

token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)
pos_embedding_layer   = torch.nn.Embedding(max_length, output_dim)   #A

dataloader = create_dataloader_v1(
  raw_text, batch_size=8, max_length=max_length, stride=max_length)
inputs, targets = next(iter(dataloader))          # inputs: (8, 4)

token_embeddings = token_embedding_layer(inputs)                    #B
# shape: (8, 4, 256)   -> (batch, tokens, emb_dim)

pos_embeddings = pos_embedding_layer(torch.arange(max_length))      #C
# shape: (4, 256)

input_embeddings = token_embeddings + pos_embeddings               #D
# shape: (8, 4, 256)

2.9 The full input pipeline

That completes the journey from raw string to model input. Every arrow below is a step this chapter built:

Chapter 2 in one breath: raw text → tokens → token IDs (via BPE) → token embeddings (a learned lookup) → + positional embeddings = the input tensor. LLMs can’t read text, so we turned it into continuous vectors compatible with neural-network math.