Classification Fine-Tuning: Task & Data · Build an LLM From Scratch

Stage 3 begins. The foundation model understands language; now we specialize it. This chapter takes the classification path from chapter 1’s roadmap — step 8, fine-tuning the pretrained LLM into a spam classifier — and this unit sets up the task and its data.

6.1 Two kinds of fine-tuning

The most common ways to fine-tune are instruction fine-tuning — training on tasks described in natural-language prompts — and classification fine-tuning , this chapter’s topic. If you’ve done classic machine learning, the latter is familiar territory: like training a convolutional network to classify handwritten digits, just with a GPT as the backbone.

Aspect⇅	Classification fine-tuning (ch6)⇅	Instruction fine-tuning (ch7)⇅
What it produces	A specialist: fixed classes only	A generalist: follows varied instructions	+
Best suited for	Precise categorization — spam detection, sentiment analysis	Handling a variety of tasks from complex user instructions	+
Data needed	Less — labeled (text, class) pairs	More — diverse (instruction, response) pairs	+
Compute needed	Less (this chapter: minutes on a laptop)	More	+
Output interface	2-way logits from a replaced head	Free-form generated text	+

Choosing the right approach — click a row for the trade-off detail.

6.2 Preparing the dataset

The raw material: the UCI SMS Spam Collection — real text messages labeled spam / ham (non-spam) — downloaded, processed with pandas, and split three ways:

</> Dataset source and the three splits (as given in the notes)

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
# => download, extract, load into a DataFrame, then split:
# train_df.to_csv("train.csv", index=None)
# validation_df.to_csv("validation.csv", index=None)
# test_df.to_csv("test.csv", index=None)

Training data teaches; validation data steers choices during training (ch5’s lesson); the test set is touched only once, at the very end, for an unbiased estimate of real-world accuracy.

6.3 Creating data loaders

Unlike pretraining’s endless token stream, SMS messages are short, variably sized, and each carries a label. SpamDataset handles all three facts — pre-tokenize, optionally apply truncation , then apply padding to bring everything to one length:

</> SpamDataset — tokenize once, truncate, pad, pair with labels

import torch
from torch.utils.data import Dataset

class SpamDataset(Dataset):
  def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):  #A
      self.data = pd.read_csv(csv_file)

      # Pre-tokenize texts
      self.encoded_texts = [
          tokenizer.encode(text) for text in self.data["Text"]
      ]

      if max_length is None:
          self.max_length = self._longest_encoded_length()  #B
      else:
          self.max_length = max_length
          # Truncate sequences if they are longer than max_length
          self.encoded_texts = [
              encoded_text[:self.max_length]
              for encoded_text in self.encoded_texts
          ]

      # Pad sequences to the longest sequence
      self.encoded_texts = [
          encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))  #C
          for encoded_text in self.encoded_texts
      ]

  def __getitem__(self, index):
      encoded = self.encoded_texts[index]
      label = self.data.iloc[index]["Label"]
      return (
          torch.tensor(encoded, dtype=torch.long),
          torch.tensor(label, dtype=torch.long)  #D
      )

  def __len__(self):
      return len(self.data)

  def _longest_encoded_length(self):
      max_length = 0
      for encoded_text in self.encoded_texts:
          encoded_length = len(encoded_text)
          if encoded_length > max_length:
              max_length = encoded_length
      return max_length

train_dataset = SpamDataset(csv_file="train.csv", max_length=None,
                          tokenizer=tokenizer)
val_dataset = SpamDataset(csv_file="validation.csv",
                        max_length=train_dataset.max_length,  #E
                        tokenizer=tokenizer)
test_dataset = SpamDataset(csv_file="test.csv",
                         max_length=train_dataset.max_length,
                         tokenizer=tokenizer)

#A The pad token is 50256 — <|endoftext|> from ch2, moonlighting as padding. No new vocabulary needed.
#B With max_length=None (the training set), the cap becomes the LONGEST encoded message in the data.
#C Every sequence is extended with pad tokens to exactly max_length — the batch can now stack into one rectangular tensor.
#D Each item is a supervised pair: (padded IDs, class label 0/1). Compare ch2: there the target was the input shifted by one; here it's ONE label per sequence.
#E Validation and test reuse the TRAINING set's max_length so all three splits share identical geometry.

Walk one message through the pipeline:

A labeled message

The CSV pairs each SMS text with its class: 1 = spam, 0 = ham (non-spam).

1 / 5

From SMS text to a training item. Every message — long or short — leaves the pipeline with the same shape.

One batch as the model sees it: a rectangular (8 × 120) ID tensor plus an (8,) label vector — trailing 50256s mark the padding.

</> The data loaders

from torch.utils.data import DataLoader

num_workers = 0
batch_size = 8
train_loader = DataLoader(
  dataset=train_dataset,
  batch_size=batch_size,
  shuffle=True,
  num_workers=num_workers,
  drop_last=True,  #A
)
# val_loader / test_loader: same, without shuffling

#A Identical machinery to ch2/ch5 — only the Dataset changed. drop_last discards a ragged final batch during training.

Key idea — same loaders, new contract

Pretraining paired every window with its shifted self; classification pairs a whole (padded) sequence with one label. Everything downstream of the Dataset — batching, shuffling, the training-loop skeleton — carries over from chapter 5 untouched. What must change is the model’s output: turning a 50,257-way next-token scorer into a 2-way spam detector is the next unit’s surgery.

📝 Check yourself: fine-tuning setup & data

0 / 5

1.When is classification fine-tuning the right choice over instruction fine-tuning?
2.Why must every message in a batch be padded to the same length?
3.Why do the validation and test datasets use the TRAINING set's max_length?
4.Predict: what does train_dataset[0] return?
5.In the batch figure, label 1 means "spam" and 0 means "ham". What does the model ultimately have to learn?