§2.11–2.12The Determinant · Example: Principal Components Analysis

§2.11 The determinant

The determinant determinant det(A) = product of eigenvalues; |det| measures how much A expands or contracts volume. defined in ch. 2 — open in glossary $\det(\boldsymbol{A})$ maps a square matrix to a scalar — and it is simply the product of all the eigenvalues. Its absolute value measures how much multiplication by $\boldsymbol{A}$ expands or contracts volume:

• $\det = 0$: space is contracted completely along at least one dimension — all volume is lost (the matrix is singular);
• $|\det| = 1$: the transformation preserves volume.

You have already seen this: in the EigenLens of the previous section, det = λ₁λ₂ is exactly the area of the image ellipse relative to the unit circle — set λ₂ = 0 there and watch the volume vanish.

§2.12 Example: PCA from linear algebra alone

The chapter’s payoff: deriving a real machine learning algorithm — principal components analysis principal components analysis Lossy linear compression: encode c = Dᵀx, decode Dc; optimal D = top-l eigenvectors of XᵀX. defined in ch. 2 — open in glossary — using only what these pages built.

The setup. We have $m$ points $\{\boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)}\} \subset \mathbb{R}^n$ and want lossy compression: store them with less memory, losing as little precision as possible. Encode each point as a lower-dimensional code $\boldsymbol{c}^{(i)} \in \mathbb{R}^l$ ($l < n$) with an encoder $f(\boldsymbol{x}) = \boldsymbol{c}$ and decoder $\boldsymbol{x} \approx g(f(\boldsymbol{x}))$ — the same encoder/decoder shape as chapter 1’s autoencoder autoencoder An encoder plus decoder trained to preserve information while giving the representation useful properties. defined in ch. 1 — open in glossary , but purely linear. PCA is defined by its decoder choice: matrix multiplication,

(Orthonormal columns kill two problems at once: without unit norm, any scaling of $\boldsymbol{D}_{:,i}$ could be undone by shrinking $c_i$ — infinitely many equivalent solutions; and orthogonality will make the encoder trivial. Note $\boldsymbol{D}$ still isn’t technically “an orthogonal matrix” unless $l = n$.)

Step 1 — the optimal code for a fixed decoder

Minimize the distance between a point and its reconstruction:

So the encoder is $f(\boldsymbol{x}) = \boldsymbol{D}^\top\boldsymbol{x}$ and the whole PCA reconstruction is

$r(\boldsymbol{x}) = g(f(\boldsymbol{x})) = \boldsymbol{D}\boldsymbol{D}^\top\boldsymbol{x}.$

Step 2 — the optimal directions

$\boldsymbol{D}$ is shared by all points, so now the whole dataset matters. Stack the points into the design matrix design matrix X ∈ ℝ^{m×n} stacking one example per row: X_{i,:} = (x⁽ⁱ⁾)ᵀ. defined in ch. 2 — open in glossary $\boldsymbol{X} \in \mathbb{R}^{m \times n}$, $\boldsymbol{X}_{i,:} = \boldsymbol{x}^{(i)\top}$, and (for $l = 1$, where $\boldsymbol{D}$ is a single unit vector $\boldsymbol{d}$):

See the arg-min with your hands

Linear algebra is the first of the fundamental mathematical disciplines this book rests on. The other one, ubiquitous in machine learning, is probability theory — chapter 3.