REINFORCE · Directed Generative Nets · Sigmoid Belief Nets · Differentiable Generators

§20.9.1 REINFORCE — gradients through discrete samples

The reparametrization trick needs continuous $y$ . For a discrete sample, $y = f(z;\omega)$ is a step function — derivatives are zero almost everywhere, so back-propagation gets no signal. But the expected cost $\mathbb{E}_{z}[J(f(z;\omega))]$ is usually smooth, and its gradient has a tidy form:

The REINFORCE estimator (eq 20.62)

\frac{\partial\, \mathbb{E}[J(y)]}{\partial \omega} = \mathbb{E}_{y}\!\left[J(y)\,\frac{\partial \log p(y)}{\partial \omega}\right] \approx \frac{1}{m}\sum_{i} J(y^{(i)})\,\frac{\partial \log p(y^{(i)})}{\partial \omega}

$J(y)$	the cost of a sampled outcome — used only as a scalar WEIGHT, never differentiated (so J may be a black box / non-differentiable)	scalar
$\frac{\partial \log p(y)}{\partial \omega}$	the score of the sampling distribution — differentiable w.r.t. the parameters ω, even when y itself is discrete	gradient
$\frac{1}{m}\sum_i$	an UNBIASED Monte Carlo estimate: draw samples y⁽ⁱ⁾, weight each score by its cost. Correlating good outcomes with high scores nudges ω toward them	average

Cheap but noisy — the need for a baseline

REINFORCE works by correlating which $y$ was sampled with how good $J(y)$ turned out — but if a good $y$ is unlikely under the current parameters, it takes many samples to stumble on it, so the estimator has very high variance. The fix: subtract a baseline $b(\omega)$ from $J(y)$ . Because $\mathbb{E}_y[\partial\log p(y)/\partial\omega] = 0$ , any $b$ that doesn’t depend on $y$ leaves the gradient’s expectation unchanged (eqs 20.66–20.67) while shrinking its variance — often a learned estimate of the average cost.

Two ways to back-propagate through sampling. Click a cell for detail.
	Method	When it applies	Character
Reparametrization trick	reparametrization (§20.9)	CONTINUOUS y	low variance, exact pathwise gradient
REINFORCE	REINFORCE (score-function)	DISCRETE y (or non-differentiable J)	unbiased but high variance; needs a baseline

Dotted-underlined cells have explanations — click one.

§20.10 Directed generative nets

Within deep learning, directed models were long overshadowed by the RBM — until around 2013, when differentiable generators (VAEs, GANs) brought them to the fore.

§20.10.1 Sigmoid belief networks

A sigmoid belief network is a directed model of binary units, each a sigmoid of its ancestors:

The sigmoid belief network conditional (eq 20.70)

p(\mathrm{s}_i) = \sigma\!\left(\sum_{j < i} W_{j,i}\, s_j + b_i\right)

$\sum_{j<i} W_{j,i} s_j$	input from the ANCESTORS (earlier units) — a directed acyclic structure, sampled ancestrally top-down	scalar
$\sigma(\cdot)$	each unit turns on with a sigmoid probability. Layered SBNs are UNIVERSAL approximators of distributions over the visibles	probability

Sampling is cheap (one ancestral pass, top layer down), but inference is intractable (explaining away couples the latents, and mean field’s cliques span whole layers). The classic remedy is a learned inference net — the Helmholtz machine pairs an SBN with a network predicting the mean-field parameters, trained by wake-sleep. But the discrete latents block the reparametrization trick, so training leans on the noisier REINFORCE machinery (modern reweighted wake-sleep and bidirectional Helmholtz machines have made this practical). A special case with no latents is the fully-visible belief network (§20.10.7).

§20.10.2 Differentiable generator networks

The unifying modern idea: a differentiable generator network $g(\boldsymbol{z};\boldsymbol{\theta})$ — a neural net that transforms latent samples $\boldsymbol{z}$ into data $\boldsymbol{x}$ (or into $p(\boldsymbol{x}\mid\boldsymbol{z})$ ):

A differentiable generator: sample simple noise z, push it through a deterministic net g to produce a sample x (or the parameters of p(x|z)). g reshapes the simple z-distribution into a complex x-distribution. This is the shared engine of VAEs and GANs.

Formally $g$ is a nonlinear change of variables reshaping $p(\boldsymbol{z})$ into $p(\boldsymbol{x})$ :

Generator as a change of variables (eq 20.73)

p_x(\boldsymbol{x}) = \frac{p_z(g^{-1}(\boldsymbol{x}))}{\left| \det\!\left(\frac{\partial g}{\partial \boldsymbol{z}}\right) \right|}

$p_z(g^{-1}(\boldsymbol{x}))$	the density of the latent that maps to x — requires g to be invertible	density
$\left\|\det(\partial g/\partial \boldsymbol{z})\right\|$	the Jacobian volume factor — usually intractable to evaluate, so we train g by INDIRECT criteria (VAE lower bound, GAN game) instead of max-likelihood on this formula	scalar

A generator can either emit $\boldsymbol{x}$ directly (continuous data only, but free of hand-designed output distributions) or emit the parameters of $p(\boldsymbol{x}\mid\boldsymbol{z})$ (e.g. sigmoid means of Bernoullis — handles discrete data). Either way, the reparametrization trick lets us train it by gradient descent. The two dominant instances — pairing $g$ with an inference net (the VAE) or with a discriminator (the GAN) — come next.

Check yourself — REINFORCE & differentiable generators

1.Why does the reparametrization trick fail for discrete samples, and what replaces it?

2.What is the main practical problem with the basic REINFORCE estimator, and how is it addressed?

3.What is the tradeoff of a sigmoid belief network?

4.What is a differentiable generator network, and why don't we train it by maximizing eq 20.73's change-of-variables likelihood directly?

5.A differentiable generator can either emit x directly or emit the parameters of p(x|z). What's the difference?

5 questions