§11.0–11.4Design Process · Performance Metrics · Baselines · Gathering Data

§11.0 A practical design process

Knowing many algorithms matters less than applying a commonplace one correctly. In practice you can usually do far better with a correct application of an everyday algorithm than a sloppy application of an exotic one. That correct application rests on a simple methodology (adapted from Ng 2015):

The running example throughout is Google’s Street View house-number transcription system — a CNN that reads address numbers from photos to place them on the map.

§11.1 Performance metrics

Your error metric guides every later decision, so choose it — and a realistic target — first. Absolute-zero error is usually impossible: the Bayes error bayes error The error an oracle knowing the true p(x,y) still makes — the irreducible noise floor. defined in ch. 5 — open in glossary is the floor even with infinite data, because the features may not fully determine the output or the task may be intrinsically stochastic. And the metric usually differs from the training cost.

Plain accuracy (§5.1.2) often isn’t enough. When mistakes have unequal costs (a spam filter blocking a real email ≫ letting spam through), optimize a total cost. For rare events, accuracy is actively misleading: precision precision (classification) Of all the detections a model reports, the fraction that are correct: TP/(TP+FP). (Distinct from the ch03 statistical precision = inverse variance.) Paired with recall for rare-event tasks where accuracy is uninformative. defined in ch. 11 — open in glossary (fraction of detections that are correct) and recall recall Of all the true positive events, the fraction the model detects: TP/(TP+FN). A detector that flags everyone has perfect recall but poor precision. Traded against precision by moving the decision threshold. defined in ch. 11 — open in glossary (fraction of true events found) tell the real story. Drag the threshold to trade one for the other, and push prevalence toward zero to watch accuracy stay near 100% while precision collapses:

Sweep the threshold and you trace a PR curve pr curve A curve of precision (y) vs recall (x) traced by sweeping the classifiers decision threshold; summarizes the precision/recall tradeoff. Often reduced to a single number via the F-score or the area under the curve (PR-AUC). defined in ch. 11 — open in glossary . To collapse it to one number, use the area under it or the F-score f-score A single-number summary of precision p and recall r: F = 2pr/(p+r) (the harmonic mean). High only when BOTH precision and recall are high; used when a rare-event detector must be summarized by one metric. defined in ch. 11 — open in glossary :

Some systems may refuse to decide when unsure and hand off to a human. Then coverage coverage The fraction of examples for which a system chooses to produce a response (rather than refuse/defer to a human). Traded against accuracy: refusing every example gives 100% accuracy at 0% coverage. Street View targets 98% accuracy at ≥95% coverage. defined in ch. 11 — open in glossary — the fraction of examples answered — is the metric, traded against accuracy: refusing everything gives 100% accuracy at 0% coverage. Street View targets human-level 98% accuracy while pushing coverage above 95%. Whatever the choice, fix the metric before iterating, or you won’t know if a change helped.

§11.2 Default baseline models

Establish a sensible end-to-end system fast. If the problem might yield to a few linear weights, start simple (logistic regression). If it’s AI-complete (object/speech recognition, translation), start with deep learning — and pick the baseline default baseline model The first sensible end-to-end system to build for a task, chosen by data structure: MLP for fixed vectors, CNN for grids/images, gated RNN (LSTM/GRU) for sequences — with ReLUs, SGD+momentum or Adam, early stopping, and dropout. defined in ch. 11 — open in glossary by the structure of your data:

Sensible defaults for the rest: optimize with SGD + momentum and a decaying learning rate, or Adam; add batch normalization if optimization struggles (especially CNNs and sigmoidal nets). Include mild regularization from the start — early stopping almost universally, plus dropout (batch norm’s noise can sometimes substitute). If a similar task has been studied, copy its best model — or use transfer learning transfer learning Initialize a network with the first k layers of a net trained on a related task/dataset, then jointly fine-tune on the target task (often with fewer examples); §15.2. defined in ch. 8 — open in glossary (e.g. ImageNet CNN features). Use unsupervised learning in the baseline only when the domain (like NLP word embeddings) is known to benefit, or the task is itself unsupervised.

§11.3 Determining whether to gather more data

Once the baseline runs, resist the urge to swap algorithms at random — gathering more data is often more effective. But first, diagnose:

How much more data? Plot generalization error against training-set size (like fig 5.4) and extrapolate. Since a small fraction of extra examples rarely moves the needle, experiment on a logarithmic scale — e.g. double the dataset between runs. If gathering much more data isn’t feasible, improving generalization becomes a research problem (better algorithms), not an application one.

§11.4 Selecting hyperparameters

Deep learning algorithms come with many hyperparameters governing runtime, memory, and model quality. There are two ways to set them: manually (requires understanding what each does and how models generalize) and automatically (less understanding needed, but far more compute). The next section digs into both — manual tuning, grid search, and random search: selecting hyperparameters.