§7.12 Dropout — bagging made practical
Bagging (§7.11) is powerful, but training and evaluating many large neural networks is expensive. Dropout dropout A cheap, powerful regularizer that approximates BAGGING over the exponentially many sub-networks formed by multiplying units by a random binary mask µ (keep-prob ≈0.8 input, 0.5 hidden). Training minimizes E_µ J(θ,µ); the sub-networks share parameters. defined in ch. 7 — open in glossary is the trick that makes it practical: to a first approximation, it is bagging over an ensemble of exponentially many networks, at almost no extra cost. The ensemble consists of every sub-network you can form by removing non-output units from a base network — and removing a unit just means multiplying its output by zero.
A base net with 2 inputs and 2 hidden units has such sub-networks. Resample the mask to visit them, or view the whole ensemble (fig 7.6):
Each minibatch example multiplies every input/hidden unit by an independent 0/1 mask (keep-prob 0.8 input, 0.5 hidden), then runs the usual forward/backward pass on the surviving sub-network. The output unit is never dropped. Over training we visit an exponential number of these networks — all sharing one set of weights.
The procedure rides on top of ordinary minibatch SGD. Each time an example enters a minibatch, sample a fresh binary mask dropout mask The per-example binary vector µ (one entry per input/hidden unit, each sampled independently with its keep-probability) that selects which units are kept in that sub-network; multiplying a unit by 0 removes it. defined in ch. 7 — open in glossary — one entry per input and hidden unit, each drawn independently with a fixed keep-probability (typically 0.8 for inputs, 0.5 for hidden units) — multiply every unit by its mask entry, and run the usual forward propagation, back-propagation, and update on the surviving sub-network. Formally, dropout minimizes the expected cost over masks:
Term by term
| the binary mask: which units are kept this step, each sampled independently with its keep-probability | the mask | |
| the cost of the single sub-network selected by θ and µ | one sub-net cost | |
| the average over ALL masks — exponentially many terms, but drawing one µ per step gives an UNBIASED estimate of the gradient | the ensemble |
Why an exponential ensemble is affordable
The exponential term count looks hopeless, but two things save it. First, we never sum over all masks — we sample one per example, and the stochastic gradient is unbiased. Second, all the sub-networks share one set of weights: each is just the parent net with some units switched off, so the entire ensemble costs the same memory as a single network. Dropout adds only the cost of generating the mask.
How dropout differs from true bagging
Dropout is bagging with two twists — parameter sharing, and the fact that almost no member is ever trained on its own:
| bagging | dropout | |
|---|---|---|
| parameters | each model has its OWN independent parameters | all sub-networks SHARE parameters — each inherits a subset from the parent net |
| training | each model is trained to convergence on its dataset | most sub-networks are never explicitly trained; a tiny fraction each take one step, and parameter sharing propagates good settings to the rest |
| data per member | a dataset resampled with replacement | the same — each sub-network sees a subset of the training set (its minibatch examples) |
There are far too many sub-networks to sample in the lifetime of the universe, so most are never touched directly — yet because they share weights, training a sliver of the ensemble tunes them all. That is the whole trick.
Next: how to make a prediction from this ensemble in a single forward pass — the weight-scaling inference rule — plus adversarial training — dropout part 2 & adversarial training.
Check yourself — dropout as bagging
1.What ensemble does dropout train, and how is a sub-network formed?
2.In DropoutLab, click 'show all 16'. Why are there exactly 16 sub-networks, and why is the output unit never among the dropped ones?
3.How is the dropout objective E_µ J(θ,µ) optimized despite having exponentially many terms?
4.What are the two key ways dropout differs from ordinary bagging?
5.Why does dropout cost roughly the same memory as a single network, despite representing an exponential ensemble?