Bias · Variance and Standard Error · The Trade-off · Consistency

§5.4.2 Bias

The first quality measure of an estimator is its bias — systematic deviation from truth:

Term by term

\mathrm{bias}(\hat{\theta}_m) = \mathbb{E}(\hat{\theta}_m) - \theta

$\mathbb{E}(\hat{\theta}_m)$	the estimator’s average over datasets — the expectation is over the data seen as random samples	over resamples
$\theta$	the true underlying value that generated the data	fixed, unknown
$\mathrm{bias}$	systematic deviation. Zero → UNBIASED; zero in the m → ∞ limit → asymptotically unbiased	error type 1

Three classics, worked. The Bernoulli sample mean is clean:

The sample mean of a Bernoulli(θ) is unbiased (eqs 5.23 → 5.28)

\hat{\theta}_m = \frac{1}{m}\sum_{i=1}^m x^{(i)}

step 1/3: The candidate estimator: average the m binary samples.

The variance estimator is the famous exception:

The sample variance is BIASED — and the m−1 fix (eqs 5.36 → 5.44)

\hat{\sigma}^2_m = \frac{1}{m}\sum_{i=1}^m \big(x^{(i)} - \hat{\mu}_m\big)^2

step 1/4: The natural candidate: average squared deviation from the SAMPLE mean.

§5.4.3 Variance and standard error

An estimator’s own variance measures how much it wobbles across independent resamplings of the dataset; its square root is the standard error $\mathrm{SE}(\hat{\theta})$ . The workhorse case:

Term by term

\mathrm{SE}(\hat{\mu}_m) = \sqrt{\mathrm{Var}\Big[\frac{1}{m}\sum_i x^{(i)}\Big]} = \frac{\sigma}{\sqrt{m}}

$\sigma/\sqrt{m}$	the mean’s uncertainty shrinks with √m — quadrupling data halves it (the Bernoulli case: Var(θ̂) = θ(1−θ)/m, same 1/m law)	shrinks slowly
$\hat{\mu}_m \pm 1.96\,\mathrm{SE}$	the 95% confidence interval, via the central limit theorem (the mean is approximately normal). ML practice: algorithm A beats B when A’s upper bound < B’s lower bound	the comparison rule
$\hat{\sigma}$	estimating σ itself: both √(sample variance) variants UNDERestimate the true standard deviation; the √(unbiased) version less so — fine for large m	caveat

§5.4.4 The bias-variance trade-off

Bias and variance are the two sources of estimator error — systematic deviation vs sampling wobble. Given one estimator of each flavor, which to prefer? Cross-validation settles it empirically; analytically, compare the mean squared error:

Term by term

\mathrm{MSE} = \mathbb{E}\big[(\hat{\theta}_m - \theta)^2\big] = \mathrm{Bias}(\hat{\theta}_m)^2 + \mathrm{Var}(\hat{\theta}_m)

$\mathrm{Bias}^2$	squared systematic error — typically FALLS as capacity rises	underfit side
$\mathrm{Var}$	sampling sensitivity — typically RISES as capacity rises	overfit side
$\mathrm{MSE}$	their sum: desirable estimators keep both in check — and its U-shape over capacity IS the generalization-error U-curve of §5.2, rederived from statistics	the U again

Fig 5.6 (recreated) — the bias-variance trade-off : the statistical anatomy of §5.2’s U-curve.

§5.4.5 Consistency

As data grows we want convergence to the truth:

Term by term

\mathrm{plim}_{m \to \infty}\, \hat{\theta}_m = \theta

$\mathrm{plim}$	convergence in probability: P(\|θ̂ − θ\| > ε) → 0 for every ε > 0 — this is (weak) CONSISTENCY; strong consistency = almost-sure convergence	limit notion
$\Rightarrow$	consistency implies the bias vanishes asymptotically…	one way
$\not\Leftarrow$	…but NOT conversely: asymptotic unbiasedness does not imply consistency	the trap

Aside — the classic counterexample

Estimate a Gaussian mean by always using the first sample: $\hat{\theta} = x^{(1)}$ . Then $\mathbb{E}[\hat{\theta}] = \mu$ for every m — perfectly unbiased, hence asymptotically unbiased. But it never converges : seeing a million more samples changes nothing. Unbiased ≠ making use of the data.

These properties beg a question the next section answers: instead of guessing estimators and checking them, is there a principle that derives good estimators? Yes — maximum likelihood.

Check yourself — estimator theory

1.Why is the plain sample variance (divide by m) biased, and what fixes it?

2.Your test-set error estimate has standard error SE. How much more data (test examples) halves SE, and why?

3.How do ML experiments typically declare 'algorithm A is better than B'?

4.MSE = Bias² + Variance. How does this connect to §5.2's U-curve?

5.θ̂ = x⁽¹⁾ (always use the first sample) is unbiased for a Gaussian mean. Is it consistent?

5 questions