Model-Based Optimization · Debugging · Street View Example

§11.4.5 Model-based hyperparameter optimization

Hyperparameter search is itself an optimization: the decision variables are the hyperparameters, the cost is validation error. Where that error is differentiable w.r.t. the hyperparameters we could follow the gradient — but usually it isn’t (too costly, or non-differentiable for discrete hyperparameters). So model-based optimization builds a surrogate — typically a Bayesian regression model estimating both the expected validation error and its uncertainty — and proposes new settings by balancing exploration (uncertain settings that might be much better) against exploitation (settings the model is confident are good). Tools include Spearmint, TPE, and SMAC.

Promising, but not yet a default

Bayesian hyperparameter optimization sometimes matches or beats human experts, but fails catastrophically on other problems — worth trying, not yet reliable. A practical weakness: most methods need a training run to complete before they learn anything, unlike a human who spots a pathological configuration early. Freeze/thaw approaches (Swersky 2014) mitigate this by pausing unpromising runs and resuming them if they later look worth it.

§11.5 Debugging strategies

When a system performs poorly it’s often unclear whether the algorithm is wrong or the implementation is buggy. Two things make ML uniquely hard to debug: we usually don’t know the intended behavior a priori (discovering it is the whole point — is 5% test error good or bad?), and models have many adaptive parts — if one breaks, the others compensate and still reach roughly acceptable performance, hiding the bug. Debugging tests therefore either use a case simple enough to predict, or exercise one part in isolation:

Debugging tests for neural nets — click a row for what it catches.
	What it catches
Visualize the model in action	view detections on images / listen to generated speech
Visualize the worst mistakes	sort by model confidence; inspect confident errors
Reason from train/test error	low train + high test → overfit or a save/load bug; both high → underfit or a defect
Fit a tiny dataset	if you can't overfit even one example, there is a software defect
Gradient checking	compare back-propagated derivatives to numerical ones
Monitor histograms	activations (saturation), gradients (vanish/explode), update-to-parameter ratio ≈ 1%

Dotted-underlined cells have explanations — click one.

Gradient checking deserves a closer look. To verify a hand-written back-prop, compare it to a numerical derivative — but the step size $\epsilon$ is a trap. Slide it and watch the error U-turn:

Fig (interactive) — finite-difference error vs step ε for f(x)=sin(x), computed in double precision. Large ε: truncation error. Tiny ε: floating-point rounding. Centered difference dips lower; complex-step never cancels.

step ε = 10^-6.0

forwardO(ε)

4.2e-7

centeredO(ε²)

2.8e-11

complex-stepno cancellation

9.0e-14

The forward & centered curves are U-shaped: too-large ε adds truncation error, too-small ε adds floating-point rounding error (subtracting nearly-equal numbers). Centered difference (O(ε²)) reaches a lower minimum than forward (O(ε)). The complex-step trick, imag(f(x+iε))/ε, never subtracts nearby values, so its error keeps falling to machine precision — usable with ε as tiny as 10⁻¹⁵⁰.

Numerical derivatives (eqs 11.6–11.9)

f'(x) \approx \frac{f(x+\epsilon)-f(x)}{\epsilon}\;\text{(forward)} \;\;\;\; \frac{f(x+\frac{\epsilon}{2})-f(x-\frac{\epsilon}{2})}{\epsilon}\;\text{(centered)} \;\;\;\; \frac{\operatorname{imag}\!\big(f(x+i\epsilon)\big)}{\epsilon}\;\text{(complex-step)}

$\frac{f(x+\epsilon)-f(x)}{\epsilon}$	forward difference — the basic estimate, with O(ε) truncation error	one-sided
$\frac{f(x+\frac{\epsilon}{2})-f(x-\frac{\epsilon}{2})}{\epsilon}$	centered difference — O(ε²) truncation, so more accurate for the same ε	two-sided
$\operatorname{imag}(f(x+i\epsilon))/\epsilon$	complex-step — NO subtraction of nearby values, so no cancellation; ε can be as tiny as 10⁻¹⁵⁰	complex
$u^{\top} g(v x)$	for a vector function g:ℝᵐ→ℝⁿ, test this scalar reduction with random u, v — one finite difference then exercises back-prop through all of g	random projection

§11.6 Example: multi-digit number recognition

The whole methodology, applied to Street View — step through the project:

The Street View transcription project, following the design methodology.

11 · Choose metrics

Maps are only useful if accurate, so require human-level 98% accuracy. Since that isn't always attainable, sacrifice coverage: hold accuracy at 98% and make COVERAGE the metric to optimize (goal ≥95%).

step 1 / 7

The lesson threaded through the whole chapter: diagnose before you act. A clearly-chosen metric, a fast baseline, and disciplined instrumentation turn “try random things” into a sequence of targeted, evidence-driven fixes.

Next, the chapter wraps up in review; then Part II closes with real-world applications (chapter 12) that put this methodology to work at scale.

Check yourself — Bayesian optimization, debugging, and the Street View example

1.How does model-based (Bayesian) hyperparameter optimization work, and what tradeoff does it manage?

2.Why are machine learning systems especially hard to debug?

3.You get high training error and want to know if it's genuine underfitting or a software bug. What quick test helps?

4.In the GradCheckLab, why do the forward and centered difference errors rise again as ε becomes very small, and what fixes it?

5.Which debugging strategy uncovered a systematic data problem in the Street View system?

6.What is the overall arc of the Street View case study as an example of the methodology?

6 questions