§E.1–E.4Parameter-Efficient Fine-Tuning with LoRA

The book’s final act revisits chapter 6’s spam task with a question production teams ask daily: must fine-tuning touch millions of weights at all? LoRA lora Low-rank adaptation: approximating a layer's weight update as a product of two small matrices (ΔW ≈ AB), trained while the original weights stay frozen. defined in ch. E — open in glossary (low-rank adaptation) — one of the most widely used parameter-efficient fine-tuning parameter-efficient fine-tuning Adapting a large model by training only a small fraction of (or small addition to) its parameters. defined in ch. E — open in glossary techniques — answers no: adapt the model through tiny bypass matrices while every pretrained weight stays frozen.

E.1 The idea: approximate the update, don’t replace the weights

Regular fine-tuning learns a full update matrix for each layer; LoRA learns a low-rank factorization of it:

$W_{\text{updated}} = W + \Delta W \qquad \xrightarrow{\;\text{LoRA}\;} \qquad W_{\text{updated}} = W + AB$

The separability trick. By the distributive law of matrix multiplication, the update never has to be merged into the weights:

$x(W + AB) = xW + xAB$

The LoRA matrices stay separate — the pretrained model is never modified, and the adapters can be applied on the fly (or swapped per task over a single frozen base model). This is what makes LoRA especially attractive in practice.

E.2–E.3 Same task, same head

The dataset is ch6’s SMS Spam Collection, and the model prep repeats ch6’s head replacement — model.out_head = torch.nn.Linear(768, 2). Everything new is in how the rest of the model adapts.

E.4 The implementation — three small classes

import math

class LoRALayer(torch.nn.Module):
  def __init__(self, in_dim, out_dim, rank, alpha):
      super().__init__()
      self.A = torch.nn.Parameter(torch.empty(in_dim, rank))
      torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))  #A
      self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))  #B
      self.alpha = alpha

  def forward(self, x):
      x = self.alpha * (x @ self.A @ self.B)  #C
      return x

class LinearWithLoRA(torch.nn.Module):
  def __init__(self, linear, rank, alpha):
      super().__init__()
      self.linear = linear  #A
      self.lora = LoRALayer(
          linear.in_features, linear.out_features, rank, alpha
      )

  def forward(self, x):
      return self.linear(x) + self.lora(x)  #B

def replace_linear_with_lora(model, rank, alpha):
  for name, module in model.named_children():
      if isinstance(module, torch.nn.Linear):
          setattr(model, name, LinearWithLoRA(module, rank, alpha))  #C
      else:
          replace_linear_with_lora(module, rank, alpha)  #D

Follow one input through the wrapped layer:

Freeze, wrap, count

total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable parameters before: {total_params:,}")
# Total trainable parameters before: 124,441,346

for param in model.parameters():
  param.requires_grad = False
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable parameters after: {total_params:,}")
# Total trainable parameters after: 0  #A

replace_linear_with_lora(model, rank=16, alpha=16)
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable LoRA parameters: {total_params:,}")
# Total trainable LoRA parameters: 2,666,528  #B

How does that number depend on the rank? Each wrapped Linear contributes $r \cdot (d_{in} + d_{out})$ parameters — summing over every Linear in this model gives exactly 166,658 · r:

The payoff

Fine-tuning runs ch6’s train_classifier_simple unchanged (AdamW, lr=5e-5, 5 epochs):