Dropout, Part 2 — Weight-Scaling Inference · Adversarial Training

§7.12b Predicting with an exponential ensemble

Training was easy — sample a mask, take a step. Inference is the hard part: a bagged ensemble must combine the votes of all its members, and here there are exponentially many. The arithmetic mean $\sum_{\boldsymbol{µ}} p(\boldsymbol{µ})\,p(y\mid\boldsymbol{x},\boldsymbol{µ})$ is intractable; you can approximate it by sampling 10–20 masks and averaging, but there is a far cheaper trick. Switch to the geometric mean of the members’ distributions (Warde-Farley et al. showed it performs comparably), and a beautiful simplification appears — the weight-scaling inference rule :

Term by term

p_{\text{ensemble}}(y \mid \boldsymbol{x}) \;\approx\; p\big(y \mid \boldsymbol{x};\ \tfrac12\boldsymbol{W}\big)

$\tfrac12\boldsymbol{W}$	evaluate ONE network with all units present, but each unit’s outgoing weights multiplied by its keep-probability (½ for the usual 0.5) — one forward pass, no sampling	scale the weights
$\text{expected input}$	the ½ makes the expected total input to each unit at TEST time match its expected input at TRAIN time, where half the units were missing on average	why it works
$\text{exact?}$	provably exact for softmax/linear and conditionally-normal outputs; only an approximation for deep nonlinear nets — but empirically it can even beat 1000-sample Monte Carlo	the caveat

For a softmax classifier the rule is exactly right. Take $P(y\mid\boldsymbol{v};\boldsymbol{d}) = \mathrm{softmax}(\boldsymbol{W}^\top (\boldsymbol{d}\odot\boldsymbol{v}) + \boldsymbol{b})$ and form the ensemble’s geometric mean:

Weight scaling is exact for softmax regression (equations 7.60–7.66)

\tilde{P}_{\text{ensemble}}(y \mid \boldsymbol{v}) = \Bigg(\prod_{\boldsymbol{d}\in\{0,1\}^n} \mathrm{softmax}(\boldsymbol{W}^\top(\boldsymbol{d}\odot\boldsymbol{v}) + \boldsymbol{b})_y\Bigg)^{\!1/2^n}

step 1/4: The (unnormalized) geometric mean over all 2ⁿ input masks d; the constant 2ⁿ factor is dropped since we renormalize at the end.

Dropout is remarkably cheap and general, but it is not free — the trade-offs:

Dropout in practice
	property
Cost	O(n) per example (generate a binary mask and multiply); inference costs the same as no dropout — you pay the ÷2 once.
Generality	Works with almost any distributed-representation model trainable by SGD — feedforward nets, RBMs, RNNs. Often more effective than weight decay or sparse-activity penalties, and combinable with them.
Needs a bigger model	It reduces effective capacity, so you must enlarge the model and train longer — the total system cost can be significant.
Weak spots	Little benefit on very large datasets (regularization matters less); less effective with few labels (<5000), where Bayesian neural nets can win.

Dotted-underlined cells have explanations — click one.

Dropout is more than noise — it is adaptive, and multiplicative

Much of dropout’s power comes from masking hidden units: dropping the unit that detects a nose erases the concept “there is a nose”, forcing the net to build redundant or alternative cues. Ordinary input noise cannot surgically remove a learned feature. The noise is also multiplicative, not additive — additive fixed-scale noise could be defeated by a ReLU unit simply growing large, but multiplication by 0/1 cannot. The stochasticity itself is neither necessary (fast dropout approximates the average analytically) nor sufficient (dropout boosting uses the same noise but trains the ensemble jointly and does not regularize) — confirming that the value is in the bagging view. Variants abound (DropConnect drops individual weights; real-valued masks $\boldsymbol{µ}\sim\mathcal{N}(1,I)$ can beat binary), and batch normalization (§8.7.1) injects its own additive+multiplicative hidden-unit noise, sometimes making dropout unnecessary.

§7.13 Adversarial training

Networks that match human accuracy on an i.i.d. test set can still be broken almost completely. Search near a correctly-classified input $\boldsymbol{x}$ for a point $\boldsymbol{x}'$ that the model classifies very differently, and you find adversarial examples : $\boldsymbol{x}'$ looks identical to $\boldsymbol{x}$ to a human, yet the network is confidently wrong (fig 7.8):

The primary cause is excessive linearity. Networks are built from linear building blocks and often behave linearly overall; a linear function with weights $\boldsymbol{w}$ changes by up to $\epsilon\lVert\boldsymbol{w}\rVert_1$ when each input moves by $\epsilon$ — huge in high dimensions:

Term by term

|\Delta s| \;\le\; \epsilon\,\lVert \boldsymbol{w} \rVert_1

$\epsilon$	the per-input perturbation, taken as ε·sign(wᵢ) so every input pushes the score the same way — individually imperceptible	tiny per input
$\lVert \boldsymbol{w} \rVert_1$	the sum of \|weights\|; in high dimensions this is large, so the score moves a lot even for tiny ε	huge in high-D
$\text{fix}$	adversarial training penalizes this sensitivity — a LOCAL CONSTANCY prior that keeps the output stable near the data	the remedy

Slide the dimension and the perturbation to watch an imperceptible nudge flip a linear classifier:

Excessive linearity: a per-input change ε moves a linear score by ε‖w‖₁ = ε·n — crossing the decision margin once the dimension n is large enough

ε = 0.0020n = 500

per-input change = ε = 0.0020 (imperceptible)

Δscore = ε·n = 1.00

crosses margin at n = 1/ε = 500

still correct ✓

Each input moves by a tiny, invisible ε — but a high-dimensional linear function sums those nudges into a change of ε‖w‖₁. With enough inputs, an imperceptible perturbation overwhelms the margin. This excessive linearity is why adversarial examples exist.

Why big nonlinear nets can resist — a local-constancy prior

Training on adversarially perturbed examples ( adversarial training ) reduces error on the clean test set by discouraging this sensitive locally-linear behavior — it injects a local constancy prior . Purely linear models like logistic regression cannot resist — they are forced to be linear. Neural nets can range from nearly linear to nearly locally constant, so they can capture real linear trends while learning to resist local perturbation. The same idea powers semi-supervised learning: at an unlabeled point, take the model’s own predicted label, find an adversarial neighbor that would flip it, and train the two to agree — virtual adversarial examples , which assume different classes lie on disconnected manifolds a small perturbation should not cross.

Next: the tangent methods that exploit the manifold hypothesis directly — tangent distance, tangent prop & manifold classifiers.

Check yourself — dropout inference and adversarial training

1.What is the weight-scaling inference rule, and why does it work?

2.For a softmax classifier, the derivation shows the dropout ensemble collapses to a single softmax with weights ½W. What does this establish?

3.Why is much of dropout's power attributed to masking HIDDEN units with MULTIPLICATIVE noise, rather than adding noise to inputs?

4.In AdversarialLab, keep ε tiny (say 0.002) and slide the dimension n up. Why does the linear classifier eventually flip?

5.How does adversarial training help, and why can neural nets benefit where logistic regression cannot?

5 questions