§12.1.5–12.2Dynamic Structure · Specialized Hardware · Computer Vision

§12.1.5 Dynamic structure

Another way to accelerate inference: give the computation dynamic structure, so a system runs only the part of the graph a given input needs. When this happens inside a network — computing only the hidden units that matter for the current input — it is called conditional computation conditional computation Dynamic structure inside a network: computing only the subset of features (hidden units) an input actually needs, so rarely-relevant components are skipped for speed. defined in ch. 12 — open in glossary . Many components are relevant only for a few inputs, so computing them on demand saves time.

Cascade of classifiers. A venerable trick for detecting a rare object: run a sequence of classifiers where cheap, high-recall early stages reject easy negatives, and only survivors reach an expensive, high-precision final stage. Abandon any example as soon as a stage rejects it. Since the object is rare, most inputs never reach the costly classifier. Drag the prevalence to see the savings:

Early stages need high recall (never wrongly reject a true object); the final stage supplies high precision. A cascade reaches high capacity either by making the later stages individually powerful, or by combining many small models. Google’s Street View pipeline is a two-step cascade — locate the address number, then transcribe it.

Mixture of experts. A neural gater picks which of several expert networks handles the input:

A soft mixture combines all experts by the α-weighted average — no compute saving. A hard mixture of experts mixture of experts A model where a gater network softmax-weights the outputs of several expert networks. A HARD mixture picks a single expert per input (saving computation); a soft mixture averages all experts (no compute saving). defined in ch. 12 — open in glossary picks a single expert per input, cutting cost (best when the gating choice is small, not combinatorial). A related idea, a switch routing a unit’s input by context, is an attention mechanism (§12.4.5). The catch across all of these: reduced parallelism — different inputs take different branches, so few operations stay as big matrix multiplies; CPUs lose cache coherence and GPUs must serialize divergent warps.

§12.1.6 Specialized hardware

Beyond CPUs and GPUs, hardware designers build ASICs and FPGAs (digital, analog, or hybrid) to push efficiency further — increasingly for low-power devices like phones. A key lever is precision: though general processors use 32/64-bit floats, 8–16 bits often suffice for deep nets (more is needed for training than inference), and dynamic fixed-point representations (sharing one exponent across a layer’s numbers) cut bits further. Fewer bits shrink the chip area, power, and time of the dominant operation — multiplication. Specialized hardware matters now that single-core speed has plateaued and gains come from parallelism.

§12.2 Computer vision

Vision is effortless for humans yet hard for computers, and it has long been deep learning’s flagship application. Most work is object recognition or detection in some form:

Vision needs relatively little preprocessing — mostly standardizing pixel range and size. The next section covers what preprocessing is used (contrast normalization, augmentation) and moves on to speech recognition: preprocessing and speech recognition.