Coding an LLM Architecture: The GPT Backbone

Chapters 2 and 3 built the ingredients at toy scale — 3-dimensional embeddings, 6-token sentences — so every number fit on screen. This chapter assembles them into the real thing: the architecture of the smallest GPT-2 model (124 million parameters), and by §4.7 it will generate text. This is Stage 1, step 3 of the book’s roadmap — the last step before pretraining.

The chapter follows a seven-part plan (the book’s recurring mental model). In this section we build part 1, the GPT backbone: a complete skeleton with placeholders where the yet-unbuilt components go.

The build plan (book pp. 50/55/60): start with a backbone whose placeholders show the overall structure, implement building blocks 2–5, combine them into the transformer block (§4.5), and assemble the final architecture (§4.6).

The configuration of GPT-2 small

Every module in this chapter is parameterized by one Python dictionary. Learn these seven numbers — they recur constantly from here to appendix E:

</> Configuration of the small GPT-2 model

GPT_CONFIG_124M = {
  "vocab_size": 50257,     # Vocabulary size
  "context_length": 1024,  # Context length
  "emb_dim": 768,          # Embedding dimension
  "n_heads": 12,           # Number of attention heads
  "n_layers": 12,          # Number of layers
  "drop_rate": 0.1,        # Dropout rate
  "qkv_bias": False        # Query-Key-Value bias
}

Key⇅	Value⇅	Meaning⇅
vocab_size	50257	BPE vocabulary size	+
context_length	1024	Max tokens per input	+
emb_dim	768	Embedding dimension	+
n_heads	12	Attention heads	+
n_layers	12	Transformer blocks	+
drop_rate	0.1	Dropout rate	+
qkv_bias	0	Q/K/V bias terms (False)	+

What each configuration key controls — click a row for the connection to earlier chapters.

The GPT model, zoomed all the way out

Before any code: the whole machine in one picture, with the chapters that build each piece. The goal is to complete the sentence “Every effort moves you” → “forward”, one token at a time:

Recreation of the book’s p. 50 overview: tokenized text → embedding layers → repeated transformer blocks (with ch3’s masked multi-head attention inside) → output layers → the next word.

A placeholder skeleton: DummyGPTModel

The book builds top-down: first a complete but hollow model whose placeholder blocks pass data through unchanged. That pins down the module structure and every tensor shape; sections 4.2–4.6 then replace the dummies with the real components.

</> DummyGPTModel — the skeleton with placeholder blocks

import torch
import torch.nn as nn

class DummyGPTModel(nn.Module):
  def __init__(self, cfg):
      super().__init__()
      self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["emb_dim"])  #A
      self.pos_emb = nn.Embedding(cfg["context_length"], cfg["emb_dim"])
      self.drop_emb = nn.Dropout(cfg["drop_rate"])

      # Use a placeholder for TransformerBlock
      self.trf_blocks = nn.Sequential(
          *[DummyTransformerBlock(cfg) for _ in range(cfg["n_layers"])])  #B

      # Use a placeholder for LayerNorm
      self.final_norm = DummyLayerNorm(cfg["emb_dim"])
      self.out_head = nn.Linear(
          cfg["emb_dim"], cfg["vocab_size"], bias=False
      )  #C

  def forward(self, in_idx):
      batch_size, seq_len = in_idx.shape
      tok_embeds = self.tok_emb(in_idx)
      pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))  #D
      x = tok_embeds + pos_embeds
      x = self.drop_emb(x)
      x = self.trf_blocks(x)
      x = self.final_norm(x)
      logits = self.out_head(x)
      return logits

class DummyTransformerBlock(nn.Module):
  def __init__(self, cfg):
      super().__init__()

  def forward(self, x):
      return x  #E

class DummyLayerNorm(nn.Module):
  def __init__(self, normalized_shape, eps=1e-5):
      super().__init__()

  def forward(self, x):
      return x  #E

#A The ch2 embedding pipeline at GPT-2 scale: token embeddings (50257×768) + positional embeddings (1024×768).
#B 12 placeholder transformer blocks in an nn.Sequential — the real TransformerBlock replaces these in §4.6.
#C The output head projects each 768-D vector to 50,257 vocabulary scores. bias=False, and note it is NOT tied to tok_emb here (that matters for parameter counting in §4.6).
#D torch.arange(seq_len) generates positions 0…seq_len−1; device=in_idx.device keeps the lookup on the same device as the inputs.
#E The placeholders do nothing — they exist so the data flow and interfaces are already correct.

Feed it two real sentences, tokenized with ch2’s BPE tokenizer:

</> Tokenizing a 2-sentence batch and running the skeleton

import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
batch = []
txt1 = "Every effort moves you"
txt2 = "Every day holds a"
batch.append(torch.tensor(tokenizer.encode(txt1)))
batch.append(torch.tensor(tokenizer.encode(txt2)))
batch = torch.stack(batch, dim=0)
print(batch)
# tensor([[6109, 3626, 6100,  345],  #A
#         [6109, 1110, 6622,  257]])

torch.manual_seed(123)
model = DummyGPTModel(GPT_CONFIG_124M)
logits = model(batch)
print("Output shape:", logits.shape)
# Output shape: torch.Size([2, 4, 50257])  #B

#A Both sentences start with token 6109 ("Every") — same ID, same embedding row. Four BPE tokens each.
#B The shape story of the whole model: 2 sequences × 4 positions × 50,257 vocabulary scores per position.

batch (2×4 token IDs)

"Every effort moves you"	6109	3626	6100	345
"Every day holds a"	6109	1110	6622	257
	pos 0	pos 1	pos 2	pos 3

The input batch. Note both rows share token 6109 at position 0 — the embedding stage will give them identical vectors; only the transformer blocks' context-mixing tells them apart.

Follow the shapes through the skeleton

The single most useful habit for the rest of the book: track the tensor shape at every stage. Step through the forward pass:

Token IDs in

The input is a (batch_size=2, seq_len=4) integer tensor — the BPE token IDs from ch2. No vectors yet.

1 / 6

The DummyGPTModel forward pass, shape by shape. The (2, 4, 768) shape survives untouched from the embedding sum to the final norm — transformer blocks are shape-preserving.

Key idea — a GPT is a next-token scoring machine

Whatever happens inside, the interface is fixed: token IDs in (batch, seq_len) → logits out (batch, seq_len, vocab_size). Every position gets a raw score for each of the 50,257 possible next tokens. Training (ch5) shapes those scores; generation (§4.7) picks from them. The placeholders make one more thing obvious: everything between the embeddings and the output head — the entire transformer block stack — is a shape-preserving transformation of (batch, seq_len, 768).

What’s missing from the skeleton is exactly the chapter’s to-do list: §4.2–4.3 build layer normalization and the GELU feed-forward network, §4.4–4.5 add shortcut connections and assemble the real transformer block, and §4.6 swaps everything in.

📝 Check yourself: the GPT backbone

0 / 5

1.In GPT_CONFIG_124M, what does "emb_dim": 768 control?
2.Predict: the DummyGPTModel processes a batch of two 4-token sentences. What is the output (logits) shape?
3.Why does the chapter first build a DummyGPTModel with placeholder blocks that just return their input?
4.What role does the positional embedding play in the forward pass tok_embeds + pos_embeds?
5.The two input sentences both start with token 6109 ("Every"). What does the model see for these two positions after the embedding stage?