§1.0The Deep Learning Idea

Inventors have dreamed of machines that think since antiquity — Pygmalion, Daedalus and Hephaestus were legendary inventors, and Galatea, Talos and Pandora artificial life. When programmable computers were first conceived, over a century before one was built, Ada Lovelace was already wondering whether they might become intelligent. Today artificial intelligence (AI) artificial intelligence The field of building software that solves tasks requiring intelligence. defined in ch. 1 — open in glossary is a thriving field that automates routine labor, understands speech and images, makes medical diagnoses and supports science.

The hard problems are the intuitive ones

Early AI rapidly conquered problems that are intellectually hard for humans but easy to state formally. IBM’s Deep Blue beat the world chess champion in 1997 — but chess is a tiny formal world: sixty-four squares, thirty-two pieces, rigidly circumscribed moves, all describable by a brief list of rules supplied in advance. The true challenge turned out to be the opposite kind of task: things people do intuitively and cannot describe formally — recognizing a spoken word, or a face in an image. Your everyday life runs on an immense amount of subjective, informal knowledge about the world, and getting that knowledge into a computer is one of the key challenges of AI.

The first attempt was to hand-write the knowledge: the knowledge base approach knowledge base approach Hard-coding world knowledge in formal languages for automatic logical inference (e.g. Cyc). defined in ch. 1 — open in glossary , where humans enter formal statements and an inference engine reasons over them logically.

The failures of hard-coded knowledge suggest AI systems must acquire their own knowledge from raw data — that capability is machine learning machine learning Acquiring knowledge by extracting patterns from raw data instead of hard-coding it. defined in ch. 1 — open in glossary . It let computers make apparently subjective real-world decisions: logistic regression can recommend whether to deliver by cesarean; naive Bayes can separate legitimate e-mail from spam.

Representation decides everything

Simple learners depend entirely on the representation of the data they receive. When logistic regression recommends cesarean delivery, it never examines the patient — a doctor feeds it relevant pieces of information, such as the presence or absence of a uterine scar. Each such piece is a feature feature One piece of information included in the representation of the data given to a learner. defined in ch. 1 — open in glossary . The algorithm learns how each feature correlates with outcomes, but it cannot influence how the features are defined. Given raw MRI pixels instead of the doctor’s report, it would be useless: an individual pixel has negligible correlation with delivery complications.

This dependence on representation is everywhere in computer science and in daily life: a well-indexed data collection is searched exponentially faster; arithmetic is quick on Arabic numerals and painful on Roman ones. The figure below is the book’s fig. 1.1, made live — the same data in two coordinate systems:

Separating the classes with a straight line is impossible in Cartesian coordinates and trivial in polar coordinates. Nothing about the data changed — only its representation.

So one recipe for AI is: design the right features by hand, then hand them to a simple learner. A useful feature for identifying a speaker from sound is an estimate of the size of their vocal tract — it strongly cues whether the speaker is a man, woman or child. But for many tasks we do not know what features to extract. To detect cars, we would like a “wheel” feature — yet a wheel in pixels is a simple geometric shape complicated by shadows, sun glare off the metal, occluding fenders and foreground objects. Nobody can write that detector down.

Learning the representation itself

Representation learning representation learning Using learning to discover not just representation→output but the representation itself. defined in ch. 1 — open in glossary uses machine learning to discover not only the mapping from representation to output but the representation itself. Learned representations beat hand-designed ones, and they let AI systems adapt to new tasks with minimal human effort — minutes for a simple task, hours to months for a complex one, versus decades of an entire research community for hand-engineering.

The quintessential representation learner is the autoencoder autoencoder An encoder plus decoder trained to preserve information while giving the representation useful properties. defined in ch. 1 — open in glossary : an encoder converts the input into a new representation, a decoder converts it back. It is trained to preserve as much information as possible through the round trip, while the representation is also shaped to have useful properties (different autoencoder families target different properties).

What should a good representation separate? The factors of variation factors of variation The separate sources of influence that explain the observed data, often unobserved. defined in ch. 1 — open in glossary that explain the data — separate sources of influence, usually not directly observable: sometimes unobserved objects or forces in the world, sometimes constructs in the human mind that simplify explanation. For a speech recording the factors include the speaker’s age, sex, accent and the words spoken; for a car image, the car’s position, its color, and the angle and brightness of the sun. The real-world difficulty is that every factor influences every observation — a pixel of a red car is nearly black at night; the silhouette depends on viewing angle. Applications must disentangle the factors and discard the nuisance ones.

But here is the catch: extracting a factor as abstract as “accent” requires nearly human-level understanding of the data. When obtaining a representation is as hard as the original problem, representation learning seems not to help at all.

Depth: representations built from representations

Deep learning deep learning Machine learning via representations expressed in terms of simpler representations — a deep concept hierarchy. defined in ch. 1 — open in glossary resolves this by expressing representations in terms of other, simpler representations — complex concepts built out of simple ones. The book’s fig. 1.2 (recreated below) shows the canonical picture: from pixels, a first layer finds edges by comparing neighboring brightness; from edges, corners and contours; from contours, object parts; from parts, object identity.

The mapping from pixels directly to object identity is hopelessly complicated; a series of nested simple mappings, one per layer, is learnable. The input layer is visible because we observe it; the hidden layers hidden layer An intermediate layer whose targets the data does NOT specify — the learning algorithm decides what it computes. (First introduced ch01.) defined in ch. 6 — open in glossary are “hidden” because their values are not given in the data — the model itself must decide which concepts are useful for explaining the observations.

The quintessential deep model is the multilayer perceptron (MLP) multilayer perceptron A function composed of many simpler functions; the quintessential deep model (feedforward deep network). defined in ch. 1 — open in glossary , a feedforward deep network: just a mathematical function, formed by composing many simpler functions, each composition providing a new representation.

There is a second, complementary perspective: depth depth The length of the composed-function chain — the source of the name 'deep learning'. (First introduced ch01.) defined in ch. 6 — open in glossary lets the computer learn a multi-step program. Each layer is the state of memory after another set of parallel instructions; deeper networks run longer instruction sequences, and later instructions can refer back to earlier results. On this view, not everything in a layer’s activations must encode a factor of variation — part of the representation is program state, like a counter or pointer, organizing the processing rather than describing the input.

Measuring depth — it depends on the language

There are two main ways to measure depth. The first: the number of sequential instructions to evaluate the model — the longest path through the flowchart of the computation. But just as one program has different lengths in different languages, the same model has different depths depending on which operations count as single steps. The book’s fig. 1.3 uses logistic regression, $\sigma(\boldsymbol{w}^\top \boldsymbol{x})$ — toggle the “language” below:

The second way to measure depth, used by deep probabilistic models, is the depth of the concept graph rather than the computation graph. The two can differ wildly: an AI system seeing a face with one eye in shadow may first only find one eye, then — having detected the face — infer the second eye. The concept graph has just two layers (eyes, faces), but if each concept is refined given the other $n$ times, the computation graph has $2n$ layers.

Because neither view is uniquely correct, and people choose different sets of primitive elements, there is no single correct value for the depth of a model — just as there is no single correct length for a computer program — and no consensus on how deep counts as “deep.” Deep learning is safely defined as the study of models composing more learned functions or concepts than traditional machine learning does.

To summarize: deep learning is an approach to AI — a kind of machine learning that achieves power and flexibility by representing the world as a nested hierarchy of concepts, each defined in relation to simpler ones, the more abstract computed from the less abstract. How it relates to the neighboring disciplines (the Venn diagram of fig. 1.4 and the flowcharts of fig. 1.5) is the subject of the next section.