Deep learning is a specific kind of machine learning machine learning Acquiring knowledge by extracting patterns from raw data instead of hard-coding it. defined in ch. 1 — open in glossary , and this chapter is the brief course in ML that the rest of the book stands on. ML here is applied statistics with more computer and fewer confidence intervals: estimating complicated functions statistically, with less emphasis on proving intervals around them. (Comfortable already? The book invites you to skip to §5.11 — the part explaining why deep learning had to happen.)
§5.1 What is a learning algorithm?
A learning algorithm learning algorithm Mitchell: a program that improves at tasks T, measured by P, with experience E — task, measure, and data define the problem. defined in ch. 5 — open in glossary learns from data. Mitchell (1997) makes that circular-sounding sentence precise:
The T · P · E definition
“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” — Learning is not the task; learning is the means of attaining the ability to perform the task. Walking is the task; learning to walk is how the robot gets there.
Fix any two and the third defines the problem: the same T with a different E (labels vs none) is a different learning problem.
§5.1.1 The task, T
Tasks are described by how the system should process an example — a vector of features feature One piece of information included in the representation of the data given to a learner. defined in ch. 1 — open in glossary measured from some object or event (for an image: its pixel values). The book’s tour of what ML can do — click any row for its story:
| Input → output | Flagship example | |
|---|---|---|
| Classification | x → one of k categories | object recognition |
| Classification, missing inputs | partial x → category | medical diagnosis |
| Regression | x → real number | insurance claims |
| Transcription | raw data → text sequence | Street View OCR |
| Machine translation | sequence → sequence | English → French |
| Structured output | x → interdependent values | image captioning |
| Anomaly detection | events → unusual ones | card-fraud detection |
| Synthesis & sampling | (seed) → new examples | speech synthesis |
| Imputation | partial x → filled x | survey gap-filling |
| Denoising | corrupted x̃ → clean x | image restoration |
| Density estimation | x → p(x) | the master task |
§5.1.2 The performance measure, P
P must be quantitative, and task-specific. For classification-like tasks: accuracy (proportion correct) or equivalently the error rate — the expected 0-1 loss (0 if right, 1 if wrong). For density estimation, 0-1 loss is meaningless; report the average log-probability the model assigns to examples instead.
Crucially, we care about performance on data the system has never seen — that is what deployment means — so P is evaluated on a test set kept separate from all training data.
Aside — choosing P is quietly hard
Sometimes it’s unclear what to measure: transcription — whole-sequence accuracy, or partial credit per element? regression — penalize frequent medium mistakes, or rare huge ones? The application decides. Other times the ideal quantity is impractical to measure: many of the best probabilistic models represent p(x) only implicitly, making the actual probability of a point intractable — then one designs an alternative criterion, or an approximation, that still tracks the design objectives.
Next: what the algorithm gets to see — the experience E, and the first complete learning algorithm: linear regression.
Check yourself — tasks and performance measures
1.In Mitchell's T/P/E definition, is 'learning' itself the task?
2.Why does classification with missing inputs push toward learning a probability distribution?
3.Density estimation is called the 'master task' — with a caveat. Which one?
4.Why must the performance measure P be evaluated on a separate test set?
5.For density estimation, why is accuracy the wrong P, and what replaces it?