Manual Tuning · Automatic Optimization · Grid & Random Search

§11.4.1 Manual hyperparameter tuning

To tune by hand you must understand the link between hyperparameters, training error, generalization error, and compute. The primary goal is to match the model’s effective capacity to the task. Effective capacity is limited by three things: the model’s representational capacity, the optimizer’s ability to actually minimize the training cost, and the degree of regularization. A huge model helps nothing if optimization can’t find good functions or if weight decay forbids them.

Generalization error is U-shaped in most capacity-controlling hyperparameters (recall fig 5.3): too little capacity underfits (high training error), too much overfits (large train/test gap), and the sweet spot sits between. But not every hyperparameter can traverse the whole U — weight decay ≥ 0 can only subtract capacity, so if the model underfits at zero weight decay, that knob can’t rescue it.

The learning rate is the single most important hyperparameter — if you can tune only one, tune it. Its training-error curve is U-shaped, but for a subtler reason than capacity: too large and gradient descent diverges; too small and training stalls (or gets permanently stuck):

Fig 11.1 — training error vs learning rate (log scale). Note the sharp rise once the rate exceeds its optimal value: gradient descent starts increasing the error instead of decreasing it.

Most hyperparameters can be set by reasoning about whether they increase or decrease capacity. Click any row for the caveat:

Table 11.1 — the effect of hyperparameters on model capacity.
	Adds capacity when…	Reason
Number of hidden units	increased	more units → higher representational capacity
Learning rate	tuned optimally	an improper rate (too high or too low) gives low effective capacity via optimization failure
Convolution kernel width	increased	a wider kernel adds parameters
Implicit zero-padding	increased	padding before convolution keeps the representation size large
Weight-decay coefficient	decreased	smaller decay frees parameters to become larger
Dropout rate	decreased	dropping units less often lets them co-adapt to fit the training set

Dotted-underlined cells have explanations — click one.

The usual recipe for best results: a large model, regularized well (e.g. with dropout). Brute force: keep increasing capacity and training-set size until the task is solved (given the compute).

§11.4.2 Automatic hyperparameter optimization

The ideal learner takes a dataset and returns a function with no hand-tuning. Since a human tuning hyperparameters is really optimizing validation error, we can wrap the learner in a hyperparameter optimization algorithm that hides the knobs. These wrappers have their own secondary hyperparameters (e.g. the search range per hyperparameter), but those are usually far easier to set — often the same values work across many tasks.

§11.4.3 Grid search

With three or fewer hyperparameters, the common approach is grid search : pick a small finite value set per hyperparameter and train a model for every combination in their Cartesian product, keeping the best on the validation set. Choose values on a logarithmic scale (e.g. learning rate in {.1, .01, 10⁻³, 10⁻⁴, 10⁻⁵}), and repeat the search — if the best value is at a boundary, shift the grid; if it’s in the middle, zoom in. The fatal drawback: with $m$ hyperparameters each taking $n$ values, the cost is $O(n^m)$ — exponential. Even loose parallelism can’t rescue a large search.

§11.4.4 Random search

Random search is as simple to program, more convenient, and converges faster (Bergstra & Bengio 2012). Define a marginal distribution per hyperparameter and sample joint configurations — for a positive real-valued hyperparameter, a log-uniform distribution:

Log-uniform sampling (eqs 11.2–11.3)

\log\_\text{learning\_rate} \sim u(-1, -5), \qquad \text{learning\_rate} = 10^{\,\log\_\text{learning\_rate}}

$u(a,b)$	a sample from the uniform distribution on the interval (a, b)	sampler
$\log\_\text{learning\_rate}$	sample the EXPONENT uniformly, so the learning rate is uniform on a log scale (10⁻¹ … 10⁻⁵)	log space
$10^{(\cdot)}$	exponentiate to recover the actual value — do NOT bin/discretize, which would waste the extra resolution for free	value

Why does this win? When only a few hyperparameters matter, grid search wastes computation that is exponential in the number of irrelevant hyperparameters — it re-tests the same value of an influential axis for every combination of the useless ones. Random search tests a distinct value of each influential hyperparameter on nearly every trial. Toggle the two below and watch the number of distinct important-axis values explode under random search for the same budget:

Fig 11.2 (interactive) — only the horizontal hyperparameter matters. Grid spends n² trials on just n distinct important-axis values; random spends the same budget on ~n² distinct values, landing far closer to the optimum.

n = 4 (16 trials)

distinct values of the important axis tested

4 / 16 trials

best score found (peak = 1.00)

0.897

Grid search spends all 16 trials on only 4 distinct values of the important axis — every column repeats the same x for each irrelevant y, wasting effort when a hyperparameter doesn't matter.

Like grid search, random search benefits from being repeated, refining the sampling ranges based on the first run’s results. Next: model-based optimization, debugging strategies, and the full Street View case study — model-based optimization, debugging & the Street View example.

Check yourself — manual tuning, grid search, and random search

1.The goal of manual hyperparameter tuning is to match the model's EFFECTIVE capacity to the task. What three factors constrain effective capacity?

2.How does generalization error typically behave as a function of a capacity-controlling hyperparameter?

3.If you can tune only one hyperparameter, which should it be, and why?

4.According to Table 11.1, which change INCREASES model capacity?

5.What is grid search, and what is its main drawback?

6.Why does random search often beat grid search for the same budget? (See the GridRandomLab.)

6 questions