§2.1–2.4Scalars to Tensors · Matrix Multiplication · Inverses · Span

Linear algebra is continuous rather than discrete mathematics, so many computer scientists arrive with little experience of it — yet it is essential for nearly every machine learning algorithm, deep learning above all. This chapter teaches exactly the prerequisites the book needs.

§2.1 The four objects

• A scalar scalar A single number, written in italics (s ∈ ℝ), in contrast to the arrays of linear algebra. defined in ch. 2 — open in glossary is a single number, italic lowercase, introduced with its kind: “let $s \in \mathbb{R}$ be the slope,” “let $n \in \mathbb{N}$ be the number of units.”
• A vector vector An ordered 1-D array of numbers, written bold lowercase (x ∈ ℝⁿ); a point in space, one coordinate per axis. defined in ch. 2 — open in glossary is an ordered array; element $i$ is $x_i$. With $n$ real elements, $\boldsymbol{x} \in \mathbb{R}^n$ — a point in $n$-dimensional space, one coordinate per axis. Written as a column: $\boldsymbol{x} = [x_1, x_2, \dots, x_n]^\top$. Index a set of elements with a subscript set: for $S = \{1, 3, 6\}$, $x_S$ picks those three; a minus sign takes complements ($x_{-1}$ = all but $x_1$).
• A matrix matrix A 2-D array of numbers, bold uppercase (A ∈ ℝ^{m×n}); A_{i,:} is row i, A_{:,i} column i. defined in ch. 2 — open in glossary is a 2-D array, $\boldsymbol{A} \in \mathbb{R}^{m \times n}$; $A_{1,1}$ is the top-left entry, $A_{i,:}$ row $i$, $A_{:,i}$ column $i$; for expressions, $f(\boldsymbol{A})_{i,j}$ indexes the result.
• A tensor tensor An array with any number of axes on a regular grid, written sans-serif bold. defined in ch. 2 — open in glossary generalizes to any number of axes; element $(i, j, k)$ of 3-D tensor $\mathsf{\mathbf{A}}$ is $\mathsf{A}_{i,j,k}$.

The transpose transpose Mirror of a matrix across its main diagonal: (Aᵀ)_{i,j} = A_{j,i}. defined in ch. 2 — open in glossary mirrors a matrix across its main diagonal: $(\boldsymbol{A}^\top)_{i,j} = A_{j,i}$ (fig 2.1):

Element-wise arithmetic works as expected: same-shape addition ($C_{i,j} = A_{i,j} + B_{i,j}$), scalar multiply/add ($D_{i,j} = a \cdot B_{i,j} + c$). Deep learning adds one unconventional shorthand — broadcasting broadcasting DL shorthand: C = A + b adds vector b to every row of A — implicit copying. defined in ch. 2 — open in glossary : $\boldsymbol{C} = \boldsymbol{A} + \boldsymbol{b}$ means $C_{i,j} = A_{i,j} + b_j$, i.e. the vector is implicitly copied into every row rather than materializing that copy.

§2.2 Multiplying matrices and vectors

The matrix product matrix product C = AB with C_{i,j} = Σₖ A_{i,k}B_{k,j}; each entry is a row·column dot product; not commutative. defined in ch. 2 — open in glossary $\boldsymbol{C} = \boldsymbol{A}\boldsymbol{B}$ requires the inner shapes to agree — $\boldsymbol{A}$ must have as many columns as $\boldsymbol{B}$ has rows:

Step through it with real numbers:

This is not element-wise multiplication — that operation exists too and is called the Hadamard product hadamard product Element-wise matrix product A⊙B, distinct from the standard matrix product. defined in ch. 2 — open in glossary $\boldsymbol{A} \odot \boldsymbol{B}$. The dot product dot product xᵀy — matrix product of two same-length vectors; equals ‖x‖‖y‖cos θ. defined in ch. 2 — open in glossary between same-length vectors is the matrix product $\boldsymbol{x}^\top\boldsymbol{y}$ — so the matrix product computes $C_{i,j}$ as the dot product of row $i$ of $\boldsymbol{A}$ with column $j$ of $\boldsymbol{B}$.

Useful properties (a deliberately partial list):

• distributive: $\boldsymbol{A}(\boldsymbol{B} + \boldsymbol{C}) = \boldsymbol{A}\boldsymbol{B} + \boldsymbol{A}\boldsymbol{C}$
• associative: $\boldsymbol{A}(\boldsymbol{B}\boldsymbol{C}) = (\boldsymbol{A}\boldsymbol{B})\boldsymbol{C}$
• not commutative in general ($\boldsymbol{A}\boldsymbol{B} = \boldsymbol{B}\boldsymbol{A}$ does not always hold) — though the vector dot product is: $\boldsymbol{x}^\top\boldsymbol{y} = \boldsymbol{y}^\top\boldsymbol{x}$
• transpose of a product reverses the order: $(\boldsymbol{A}\boldsymbol{B})^\top = \boldsymbol{B}^\top\boldsymbol{A}^\top$

That last rule proves dot-product commutativity in one line: a scalar is its own transpose, so $\boldsymbol{x}^\top\boldsymbol{y} = (\boldsymbol{x}^\top\boldsymbol{y})^\top = \boldsymbol{y}^\top\boldsymbol{x}$.

We can now write a whole system of linear equations in three symbols:

Expanded, row by row, it is exactly the school system: $A_{1,1}x_1 + A_{1,2}x_2 + \cdots + A_{1,n}x_n = b_1$, and so on down to row $m$ — the matrix-vector notation is simply more compact.

§2.3 Identity and inverse

The identity matrix identity matrix Iₙ: ones on the main diagonal, zeros elsewhere; Iₙx = x for all x. defined in ch. 2 — open in glossary $\boldsymbol{I}_n$ changes nothing: $\forall \boldsymbol{x} \in \mathbb{R}^n,\; \boldsymbol{I}_n\boldsymbol{x} = \boldsymbol{x}$ — ones on the main diagonal, zeros elsewhere (fig 2.2 is just $\boldsymbol{I}_3$). The matrix inverse matrix inverse A⁻¹ with A⁻¹A = I; solves Ax = b analytically as x = A⁻¹b — a theoretical tool, not a numerical one. defined in ch. 2 — open in glossary is the matrix that undoes $\boldsymbol{A}$: $\boldsymbol{A}^{-1}\boldsymbol{A} = \boldsymbol{I}_n$. It solves the linear system in four mechanical steps:

§2.4 When does a solution exist? Span and independence

Think of the columns of $\boldsymbol{A}$ as directions you can travel from the origin, with $x_i$ saying how far to go along column $i$:

A linear combination linear combination Σᵢ cᵢ v⁽ⁱ⁾ — scale each vector by a coefficient and add. defined in ch. 2 — open in glossary of vectors $\{\boldsymbol{v}^{(1)}, \dots, \boldsymbol{v}^{(n)}\}$ is $\sum_i c_i \boldsymbol{v}^{(i)}$; the span span All points reachable by linear combination of a set of vectors; the span of A's columns = its column space. defined in ch. 2 — open in glossary is everything reachable that way. Solving $\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ asks: is $\boldsymbol{b}$ in the span of the columns (the column space, or range, of $\boldsymbol{A}$)? Try it:

The counting rules the widget demonstrates:

• Solutions come in 0, 1, or ∞ — never “exactly two”: if $\boldsymbol{x}$ and $\boldsymbol{y}$ both work, every blend $\alpha\boldsymbol{x} + (1-\alpha)\boldsymbol{y}$ works too.
• Solvable for every $\boldsymbol{b} \in \mathbb{R}^m$ ⟺ the column space is all of $\mathbb{R}^m$ ⟺ the columns include a set of exactly $m$ linearly independent linear independence No vector in the set is a linear combination of the others; redundant vectors add nothing to the span. defined in ch. 2 — open in glossary vectors ($n \ge m$ is necessary but NOT sufficient — two identical columns of a 2×2 span only a line). No $m$-dimensional set has more than $m$ mutually independent vectors.
• Invertibility additionally needs at most one solution per $\boldsymbol{b}$: at most $m$ columns. Together: $\boldsymbol{A}$ must be square ($m = n$) with all columns independent. A square matrix with dependent columns is singular singular matrix A square matrix with linearly dependent columns; not invertible; equivalently, some eigenvalue is zero. defined in ch. 2 — open in glossary — no inverse (though the system may still be solvable by other means).
• For square matrices the left and right inverse coincide: $\boldsymbol{A}\boldsymbol{A}^{-1} = \boldsymbol{A}^{-1}\boldsymbol{A} = \boldsymbol{I}$.