§A.1–A.4PyTorch: Tensors, Computation Graphs & Autograd

Part III 📎 Appendix pp. 128–132 · ~9 min read

  • tensor
  • computation graph
  • autograd
  • backpropagation
  • partial derivative
  • gradient

Every chapter of this book leaned on PyTorch; this appendix is the foundation course. If you worked through chapters 2–7 first, read it as a “now I see why” tour — each concept here has already earned its keep somewhere in the GPT.

A.1 What is PyTorch?

Three components, one library:

1 · Tensor libraryarray-oriented programming à la NumPy,plus GPU acceleration and a seamlessCPU ↔ GPU switch (.to(“cuda”))2 · Autograd engineautomatic differentiation over dynamiccomputation graphs — gradients withouthand-derived calculus3 · Deep-learning librarymodular building blocks: layers, losses,optimizers, pretrained models —for researchers and developers alike
The three pillars — and where this book used them: tensors everywhere (ch2+), autograd behind every loss.backward() (ch5+), nn building blocks for every module (ch3+).

A.2 Understanding tensors

A tensor generalizes the number → list → grid progression to any number of dimensions, with a NumPy-like API:

0D tensor — a scalar
2
A scalar is just a single number.
1D tensor — a vector (3 entries)
3
1
3
A vector: one axis. (Displayed as a row; the book draws it as a column — same rank.)
2D tensor — a matrix (3×4)
3
5
1
2
1
7
2
3
3
3
4
9
A matrix: rows × columns. The book's LLM tensors continue upward: (batch, tokens, emb_dim) is 3D; per-head attention scores (b, heads, T, T) are 4D.

A.3 Seeing models as computation graphs

A computation graph is a directed graph that expresses a sequence of calculations. PyTorch’s autograd system builds one in the background during every forward pass — and that recorded structure is what lets it compute gradients for backpropagation , the main training algorithm for neural networks. The book’s example graph is a one-neuron logistic classifier:

w₁trainable weightx₁input data×u = w₁ × x₁btrainable bias+z = u + ba = σ(z)ytarget labelloss = L(a, y)an intermediate resultin the computation graph
The book’s p. 130 graph: multiply, add, squash, compare. PyTorch records exactly this structure as the code runs — and can then compute the loss’s gradients with respect to the trainable parameters w₁ and b.

A.4 Automatic differentiation made easy

Backpropagation can be thought of as an implementation of the chain rule from calculus for neural networks. Two definitions first:

∂L∂w1  =  ∂L∂a⋅∂a∂z⋅∂z∂w1∇L=(∂L∂w1,  ∂L∂b)\frac{\partial L}{\partial w_1} \;=\; \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial w_1} \qquad\qquad \nabla L = \left(\frac{\partial L}{\partial w_1},\; \frac{\partial L}{\partial b}\right)

symbolmeaning
∂L/∂w1\partial L/\partial w_1a partial derivative — how fast the loss changes as ONE variable (w1w_1) changes
the product chainthe chain rule: multiply local derivatives along the graph path from the loss back to w1w_1
∇L\nabla Lthe gradient — the vector of ALL partial derivatives; exactly what gradient descent needs to update each parameter to minimize the loss

Follow the gradients flowing backward through the graph:

w₁ x₁ b u z a loss forward: z = w₁x₁ + b · a = σ(z) · loss = BCE(a, y) — every op is RECORDED ∂L/∂a ∂L/∂z = ∂L/∂a · σ′(z) ∂L/∂w₁ = ∂L/∂z · x₁ ∂L/∂b = ∂L/∂z autograd does this whole traversal automatically: loss.backward() → w₁.grad, b.grad
1

Forward pass — the graph gets recorded

Because w₁ and b were created with requires_grad=True, every operation touching them adds a node: ×, +, σ, loss. The graph is DYNAMIC — rebuilt fresh each forward pass, which is why ordinary Python control flow works inside models.

1 / 5

Backpropagation over the one-neuron graph: the forward pass records the operations; the backward pass multiplies local derivatives along each path (shown symbolically — the code below computes the actual numbers).
</> Listing A.3 — Computing gradients via autograd, both ways 
import torch.nn.functional as F
from torch.autograd import grad

y = torch.tensor([1.0])
x1 = torch.tensor([1.1])
w1 = torch.tensor([2.2], requires_grad=True)  #A
b = torch.tensor([0.0], requires_grad=True)

z = x1 * w1 + b
a = torch.sigmoid(z)
loss = F.binary_cross_entropy(a, y)

# Way 1: explicit, per-tensor
grad_L_w1 = grad(loss, w1, retain_graph=True)  #B
grad_L_b = grad(loss, b, retain_graph=True)

# Way 2: the high-level workhorse
loss.backward()  #C
print(w1.grad)
print(b.grad)
  • #A requires_grad=True is the opt-in: from here on, autograd records every operation involving this tensor. (The same flag ch3 set to False on its demo weights to reduce output clutter.)
  • #B grad(loss, w1) returns ∂loss/∂w1 explicitly. retain_graph=True keeps the recorded graph alive so a second grad() call can reuse it.
  • #C backward() traverses the whole graph once and stores every parameter's gradient in .grad — exactly what optimizer.step() reads afterward. This pairing IS the ch5/ch6/ch7 training loop's engine room.

Key idea — you write the forward pass, autograd writes the backward

Deep learning frameworks earn their keep here: define any composition of tensor operations, and the recorded computation graph plus the chain rule yield every parameter’s gradient automatically. The 12-block GPT of ch4 is just a much bigger graph — same mechanism, same one-line loss.backward(). Next: assembling real multilayer networks and the canonical training loop.

📝 Check yourself: PyTorch fundamentals

0 / 5

  1. 1.What are PyTorch's three core components?

  2. 2.In tensor terminology, what are a scalar, a vector, and a matrix?

  3. 3.What is a computation graph, and why does PyTorch build one?

  4. 4.For the graph z = x1·w1 + b, a = σ(z), loss = L(a, y): how does the chain rule produce ∂loss/∂w1?

  5. 5.What's the difference between torch.autograd.grad(loss, w1) and loss.backward()?