Maximum Likelihood Estimation · Bayesian Statistics

§5.5 Maximum likelihood estimation

Rather than guessing estimators and auditing their bias and variance, we want a principle that derives good estimators. The most common: maximum likelihood . Given examples $\mathbb{X} = \{\boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)}\}$ drawn from the unknown $p_{\mathrm{data}}$ , and a parametric family $p_{\mathrm{model}}(\boldsymbol{x}; \boldsymbol{\theta})$ :

From a product to an expectation (eqs 5.56 → 5.59)

\boldsymbol{\theta}_{\mathrm{ML}} = \arg\max_{\boldsymbol{\theta}}\, p_{\mathrm{model}}(\mathbb{X}; \boldsymbol{\theta}) = \arg\max_{\boldsymbol{\theta}} \prod_{i=1}^m p_{\mathrm{model}}(\boldsymbol{x}^{(i)}; \boldsymbol{\theta})

step 1/3: Pick the θ under which the observed data was most probable; independence factors the joint into a product.

MLE = minimizing KL = minimizing cross-entropy (eqs 5.60 → 5.61)

D_{\mathrm{KL}}(\hat{p}_{\mathrm{data}} \| p_{\mathrm{model}}) = \mathbb{E}_{\mathrm{x} \sim \hat{p}_{\mathrm{data}}}\big[ \log \hat{p}_{\mathrm{data}}(\boldsymbol{x}) - \log p_{\mathrm{model}}(\boldsymbol{x}) \big]

step 1/3: Measure the dissimilarity between the empirical distribution and the model with §3.13’s KL divergence.

What this really says

Maximum likelihood is an attempt to make the model distribution match the empirical distribution $\hat{p}_{\mathrm{data}}$ — we would prefer to match the true $p_{\mathrm{data}}$ , but only its samples are available. Training a “loss function” and matching distributions are the same act.

§5.5.1 Conditional log-likelihood and MSE

Supervised learning estimates $P(\boldsymbol{y} \mid \boldsymbol{x}; \boldsymbol{\theta})$ — the conditional ML estimator, with i.i.d. data: $\boldsymbol{\theta}_{\mathrm{ML}} = \arg\max_{\boldsymbol{\theta}} \sum_{i=1}^m \log P(\boldsymbol{y}^{(i)} \mid \boldsymbol{x}^{(i)}; \boldsymbol{\theta})$ . And the “arbitrary” MSE criterion of §5.1.4 falls out as a theorem:

Linear regression IS maximum likelihood (eqs 5.64 → 5.65)

p(y \mid \boldsymbol{x}) = \mathcal{N}\big(y;\, \hat{y}(\boldsymbol{x}; \boldsymbol{w}),\, \sigma^2\big)

step 1/3: Model the output as a Gaussian whose MEAN is the prediction — imagining many y values compatible with each x. Variance fixed by the user.

Aside — why ML is the preferred estimator

Under two conditions — the true $p_{\mathrm{data}}$ lies in the model family, and corresponds to exactly ONE θ (identifiability) — the ML estimator is consistent. Among consistent estimators it is also statistically efficient: for large m, the Cramér-Rao lower bound says no consistent estimator achieves lower parametric MSE. When m is small enough to overfit, regularizers like weight decay trade a little bias for less variance.

§5.6 Bayesian statistics

The frequentist estimates one $\boldsymbol{\theta}$ and predicts with it; the Bayesian considers all values of $\boldsymbol{\theta}$ — Bayesian inference . The worldviews swap what is random: frequentist — θ fixed, θ̂ random (a function of random data); Bayesian — the dataset is directly observed (not random), while θ is uncertain, hence a random variable with a prior $p(\boldsymbol{\theta})$ (chosen broad/high-entropy, often preferring simpler functions). Data updates it through Bayes’ rule, and predictions integrate:

Term by term

p(x^{(m+1)} \mid x^{(1)}, \dots, x^{(m)}) = \int p(x^{(m+1)} \mid \boldsymbol{\theta})\; p(\boldsymbol{\theta} \mid x^{(1)}, \dots, x^{(m)})\; d\boldsymbol{\theta}

$p(\boldsymbol{\theta} \mid x^{(1)} \dots x^{(m)})$	the posterior — typically starts high-entropy and concentrates as data arrives	weight
$\int \cdots d\boldsymbol{\theta}$	EVERY θ with positive density votes on the prediction, weighted by its posterior — remaining uncertainty flows directly into predictions, protecting against overfitting	the Bayesian move
$\text{vs. } \hat{\theta}$	the frequentist handles estimator uncertainty via its variance; the Bayesian simply integrates over it — principled (pure probability laws), but computationally costly for large m	contrast

Bayesian linear regression, worked

Gaussian prior × Gaussian likelihood → Gaussian posterior (eqs 5.71 → 5.78)

p(\boldsymbol{y} \mid \boldsymbol{X}, \boldsymbol{w}) = \mathcal{N}(\boldsymbol{y};\, \boldsymbol{X}\boldsymbol{w},\, \boldsymbol{I})

step 1/6: The likelihood: predictions Xw with unit Gaussian noise (the standard MSE setup).

Bayesian methods generalize better with limited data and cost more with lots of it. The tractable middle ground — a point estimate that keeps the prior — is MAP estimation, next.

Check yourself — MLE and Bayes

1.Why take the log in maximum likelihood?

2.'Mean squared error is a cross-entropy.' In what sense?

3.Under what conditions is the ML estimator consistent, and what extra crown does it wear?

4.What do frequentist and Bayesian statistics each treat as random?

5.In Bayesian linear regression with prior N(0, (1/α)I), what does the posterior deliver beyond weight decay's answer?

5 questions