The Experience, E · Example: Linear Regression

§5.1.3 The experience, E

What the algorithm gets to see splits machine learning into its two broad families. Unsupervised algorithms experience a dataset of features and learn useful properties of its structure — in deep learning, usually the whole distribution $p(\boldsymbol{x})$ (explicitly, as density estimation, or implicitly, for synthesis and denoising), or other roles like clustering. Supervised algorithms additionally see a label (or target) with each example — learning to predict $\boldsymbol{y}$ from $\boldsymbol{x}$ , usually by estimating $p(\boldsymbol{y} \mid \boldsymbol{x})$ . The name pictures an instructor showing the system what to do; unsupervised learning has no teacher.

The classic Iris dataset (Fisher, 1936) illustrates both: 150 plants, four measurements each (sepal length/width, petal length/width) — and a species annotation that turns it supervised.

Aside — the blurred line

The two families are not formally distinct. The chain rule ( $p(\boldsymbol{x}) = \prod_i p(x_i \mid x_1, \dots, x_{i-1})$ , eq 5.1) splits the “unsupervised” problem of modeling $p(\boldsymbol{x})$ into $n$ supervised prediction problems; and the “supervised” $p(y \mid \boldsymbol{x})$ can be recovered from unsupervised modeling of the joint via $p(y \mid \boldsymbol{x}) = p(\boldsymbol{x}, y) / \sum_{y'} p(\boldsymbol{x}, y')$ (eq 5.2). Other variants exist too: semi-supervised (some targets missing), multi-instance (only collections labeled), and reinforcement learning (a feedback loop with an environment — outside this book’s scope).

Most datasets are described by the design matrix : one example per row, one feature per column — Iris is $\boldsymbol{X} \in \mathbb{R}^{150 \times 4}$ , with $X_{i,1}$ the sepal length of plant $i$ . (When examples have different sizes — photos of different resolutions — we fall back to a set $\{\boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)}\}$ ; chs. 9–10 handle such data.) Supervised datasets add a label vector $\boldsymbol{y}$ with $y_i$ the label of example $i$ (labels can be richer — a whole sentence for speech transcription).

§5.1.4 Example: linear regression

The abstract definition, made concrete by the simplest interesting learner. Task T: predict scalar $y$ from $\boldsymbol{x} \in \mathbb{R}^n$ by outputting

Term by term

\hat{y} = \boldsymbol{w}^\top \boldsymbol{x}

$\boldsymbol{w}$	the PARAMETERS — weights deciding how each feature affects the prediction: positive weight = feature pushes ŷ up; large magnitude = large effect; zero = no effect	n×1
$\hat{y}$	the model’s prediction of y — a linear function of the input	scalar

Measure P: mean squared error on a held-out test set, $\mathrm{MSE}_{\mathrm{test}} = \frac{1}{m}\sum_i (\hat{y}^{(\mathrm{test})} - y^{(\mathrm{test})})_i^2 = \frac{1}{m}\|\hat{\boldsymbol{y}}^{(\mathrm{test})} - \boldsymbol{y}^{(\mathrm{test})}\|_2^2$ — zero exactly when predictions equal targets, growing with their Euclidean distance. Experience E: a training set $(\boldsymbol{X}^{(\mathrm{train})}, \boldsymbol{y}^{(\mathrm{train})})$ . The algorithm: minimize $\mathrm{MSE}_{\mathrm{train}}$ (justified properly by maximum likelihood in §5.5) — and for this model the minimum has a closed form:

The normal equations (eqs 5.6 → 5.12)

\nabla_{\boldsymbol{w}} \mathrm{MSE}_{\mathrm{train}} = 0

step 1/5: The MSE is a convex quadratic bowl in w — its minimum is exactly where the gradient vanishes (§4.3: solve ∇f = 0 directly instead of iterating).

Fig 5.1 (interactive) — slide w₁ along the MSE bowl, or jump straight to the normal-equations solution

w₁0.40

MSE = 0.229

red segments: the residuals whose squares the MSE averages

Aside — the bias parameter

“Linear regression” usually means the affine model $\hat{y} = \boldsymbol{w}^\top\boldsymbol{x} + b$ : still linear in the parameters, but the line no longer must pass through the origin. The intercept $b$ is called the bias parameter — the output is biased toward $b$ absent any input. (Distinct from a statistical bias, §5.4.2!) Trick: instead of adding $b$ , append a constant feature 1 to $\boldsymbol{x}$ — its weight plays the role of $b$ . The book says “linear” for affine throughout.

Linear regression is limited, but it is a complete T/P/E loop with a provably optimal learner — the reference point for every principle that follows: capacity, overfitting and underfitting.

Check yourself — experience and linear regression

1.Why does the book call the supervised/unsupervised line 'blurred'?

2.In the design matrix X ∈ ℝ¹⁵⁰ˣ⁴ for Iris, what are rows and columns — and when does the design-matrix description fail?

3.In LinRegLab you press 'solve normal equations'. What just happened, mathematically?

4.What does the 'bias parameter' b do, and how does it differ from statistical bias?

5.Which learning paradigm involves a feedback loop with an environment rather than a fixed dataset?

5 questions