Stochastic Encoders & Decoders · Denoising Autoencoders · Estimating the Score

§14.4 Stochastic encoders and decoders

Autoencoders are just feedforward networks — so the whole §6.2.2 recipe for designing output units and losses transfers directly. There, we defined an output distribution $p(\boldsymbol{y} \mid \boldsymbol{x})$ and minimized $-\log p(\boldsymbol{y} \mid \boldsymbol{x})$ ; here the input is also the target. Given a code $\boldsymbol{h}$ , treat the decoder as a conditional distribution $p_{\text{decoder}}(\boldsymbol{x} \mid \boldsymbol{h})$ and minimize $-\log p_{\text{decoder}}(\boldsymbol{x} \mid \boldsymbol{h})$ — the menu is the familiar one:

The §6.2.2 output-unit menu, applied to decoders. Click a row for the caveat.
	p_decoder(x \| h)	−log p_decoder gives
Real-valued x	Gaussian, mean from linear output units	mean squared error
Binary x	Bernoulli, parameters from sigmoid units	binary cross-entropy
Discrete x	softmax distribution	categorical cross-entropy
Correlated outputs	usually factorial (independent given h)…	cheap to evaluate

Dotted-underlined cells have explanations — click one.

The more radical step generalizes the encoder too: from a function $f(\boldsymbol{x})$ to an encoding distribution $p_{\text{encoder}}(\boldsymbol{h} \mid \boldsymbol{x})$ — both maps become noisy, their outputs sampled:

Fig 14.2 — the stochastic autoencoder: encoder and decoder are no longer functions but distributions, p_encoder(h|x) and p_decoder(x|h); each output is a sample. This is the form the variational autoencoder (§20.10.3) will inherit.

Any latent-variable model $p_{\text{model}}(\boldsymbol{h}, \boldsymbol{x})$ defines both:

Every latent-variable model is a stochastic autoencoder (eqs 14.12–14.13)

p_{\text{encoder}}(\boldsymbol{h} \mid \boldsymbol{x}) = p_{\text{model}}(\boldsymbol{h} \mid \boldsymbol{x}), \qquad p_{\text{decoder}}(\boldsymbol{x} \mid \boldsymbol{h}) = p_{\text{model}}(\boldsymbol{x} \mid \boldsymbol{h})

$p_{\text{model}}(\boldsymbol{h} \mid \boldsymbol{x})$	inference over the latents plays the encoder role	posterior
$p_{\text{model}}(\boldsymbol{x} \mid \boldsymbol{h})$	the generative conditional plays the decoder role	likelihood
$\text{compatibility}$	a trained encoder/decoder pair need NOT correspond to any single joint p_model(x, h) — but Alain et al. (2015) showed denoising training makes them compatible asymptotically, given enough capacity and examples	caveat

§14.5 Denoising autoencoders in detail

The denoising autoencoder receives a corrupted data point and must predict the original. Formally, introduce a corruption process $C(\tilde{\boldsymbol{x}} \mid \boldsymbol{x})$ — a conditional distribution over damaged versions $\tilde{\boldsymbol{x}}$ of a clean sample $\boldsymbol{x}$ :

Fig 14.3 — the DAE training graph. The clean x is corrupted to x̃ (sampled from C), encoded and decoded — and the loss compares the reconstruction against the CLEAN x, never against x̃. Typically p_decoder is factorial, its mean parameters emitted by g.

The model learns a reconstruction distribution $p_{\text{reconstruct}}(\boldsymbol{x} \mid \tilde{\boldsymbol{x}})$ from training pairs $(\boldsymbol{x}, \tilde{\boldsymbol{x}})$ :

One DAE training step

1Sample a clean example

Draw x from the training data p̂data. This is both the starting point and — crucially — the target the loss will use.

step 1 / 3

Averaged over data and corruptions, the DAE performs stochastic gradient descent on:

The DAE objective (eq 14.14)

-\ \mathbb{E}_{\boldsymbol{x} \sim \hat{p}_{\text{data}}(\boldsymbol{x})}\ \mathbb{E}_{\tilde{\boldsymbol{x}} \sim C(\tilde{\boldsymbol{x}} \mid \boldsymbol{x})}\ \log p_{\text{decoder}}\big(\boldsymbol{x} \mid \boldsymbol{h} = f(\tilde{\boldsymbol{x}})\big)

$\hat{p}_{\text{data}}$	the empirical training distribution — the outer average over clean examples	data
$C(\tilde{\boldsymbol{x}} \mid \boldsymbol{x})$	the inner average over corruptions of each example — every epoch can corrupt the same x differently	noise
$\boldsymbol{h} = f(\tilde{\boldsymbol{x}})$	the code comes from the CORRUPTED input while the log-likelihood scores the CLEAN one — undoing the damage requires knowing p_data's structure (Alain & Bengio 2013)	the trick

§14.5.1 Estimating the score

Score matching (Hyvärinen 2005; formally §18.4 territory) is an alternative to maximum likelihood: make the model’s score match the data’s at every training point. The score is a gradient field:

The score (eq 14.15)

\nabla_{\boldsymbol{x}} \log p(\boldsymbol{x})

$\nabla_{\boldsymbol{x}}$	a gradient with respect to the INPUT, not the parameters — at each point of input space, a vector	field
$\log p(\boldsymbol{x})$	the log-density; its gradient points in the direction probability increases fastest	uphill
$\nabla_{\boldsymbol{x}} \log p(\boldsymbol{x}) = \boldsymbol{0}$	at every local maximum AND minimum of the density — the zeros of the field trace the distribution's critical structure	zeros

Worked example — the score of a standard Gaussian

For $p(x) = \mathcal{N}(x; 0, 1)$ : $\log p(x) = -\tfrac{1}{2}x^2 + \text{const}$ , so the score is $\nabla_x \log p(x) = -x$ . At $x = 2$ the score is $-2$ : pointing back toward the mean, with magnitude growing the farther you stray. Learning this field everywhere is one way of learning the distribution’s structure itself.

Here is the property that makes DAEs matter far beyond denoising: with conditionally Gaussian $p(\boldsymbol{x} \mid \boldsymbol{h})$ , the DAE’s training criterion makes the autoencoder learn a vector field $g(f(\boldsymbol{x})) - \boldsymbol{x}$ that estimates the score of the data distribution (figs 14.4–14.5). The reconstruction estimates the center of mass of the clean points that could have produced $\tilde{\boldsymbol{x}}$ , so reconstruction-minus-input points toward the data manifold. Explore the exact optimal field:

Figs 14.4–14.5 (interactive) — the vector field g(f(x̃)) − x̃ of an OPTIMAL denoising autoencoder, computed exactly for a spiral data manifold under Gaussian corruption of width σ. Arrows climb toward the manifold and vanish (dots) at density maxima — on the spiral — and at the minima between its arms. Slide σ to see the fixed-noise blur tradeoff.

corruption width σ = 26

Each arrow is the optimal denoising reconstruction minus the input, g(f(x̃)) − x̃ = 𝔼[x | x̃] − x̃ — the center of mass of the clean points that could have produced x̃, minus x̃. It estimates the score ∇ₓ log p(x) (up to a factor σ²): arrows climb toward the manifold and shrink to dots where the estimated density is flat or maximal. Moderate σ: a smooth field pointing cleanly back to the spiral — zeros ON the manifold (density maxima) and between the arms (density minima), long arrows in low-probability gaps where a step uphill gains the most.

This field view is a genuine bridge between autoencoders and probabilistic models. A generic encoder-decoder — any parametrization — estimates the score when trained with:

The generic score-estimating recipe (eqs 14.16–14.17)

\lVert g(f(\tilde{\boldsymbol{x}})) - \boldsymbol{x} \rVert^2, \qquad C(\tilde{\boldsymbol{x}} \mid \boldsymbol{x}) = \mathcal{N}(\tilde{\boldsymbol{x}};\ \boldsymbol{\mu} = \boldsymbol{x},\ \boldsymbol{\Sigma} = \sigma^2 \boldsymbol{I})

$\lVert g(f(\tilde{\boldsymbol{x}})) - \boldsymbol{x} \rVert^2$	plain squared reconstruction error against the clean target	criterion
$\mathcal{N}(\tilde{\boldsymbol{x}}; \boldsymbol{x}, \sigma^2\boldsymbol{I})$	isotropic Gaussian corruption of width σ — the pairing that makes the estimator work for general f, g (Alain & Bengio 2013)	corruption
$\sigma^2$	fixed σ recovers a BLURRED distribution (regularized score matching is not consistent); consistency returns if σ → 0 as examples → ∞	the tradeoff

Three connections worth keeping

(1) Denoising a particular autoencoder (sigmoidal hiddens, linear reconstruction, Gaussian noise, MSE) is equivalent to training a Gaussian-visible RBM — a model with an explicit $p_{\text{model}}(\boldsymbol{x}; \theta)$ — by denoising score matching (Vincent 2011; the RBM arrives in §20.5.1, score matching in §18.4–18.5). (2) Score matching applied to RBMs yields reconstruction error plus a CAE-style contractive penalty (Swersky et al. 2011). (3) In general $g(f(\boldsymbol{x})) - \boldsymbol{x}$ is not guaranteed to be the gradient of anything — early results were specialized to parametrizations where it is, and Kamyshanska & Memisevic (2015) identified the family of shallow autoencoders whose field is always a legitimate score. Sampling from a trained DAE — using it as a true generative model — waits for §20.11.

Historical perspective (§14.5.1.1)

MLPs for denoising go back to LeCun (1987) and Gallinari et al. (1987), with Behnke (2001) denoising images by recurrent nets. The modern “denoising autoencoder” names something subtler: a model meant to learn a good internal representation as a side-effect of denoising (Vincent et al. 2008, 2010) — features to pretrain deeper networks with. Inayoshi & Kurita (2005) anticipated pieces (reconstruction error + hidden-layer noise in a supervised MLP), but with a linear encoder that could not learn function families as powerful as the modern DAE’s.

Next: what the two forces of autoencoder training say about manifolds, and the contractive autoencoder that grips them directly — learning manifolds & contractive autoencoders.

Check yourself — stochastic autoencoders & the score

1.For a real-valued x with a Gaussian p_decoder(x|h) whose mean comes from linear output units, minimizing −log p_decoder(x|h) gives…

2.In DAE training, what pair does one training step consume, and what is the target?

3.What is the score, and what does a DAE (Gaussian corruption, squared error) learn about it?

4.Predict the outcome — in the ScoreFieldLab, where do the arrows shrink to (near-)zero dots?

5.Training with a FIXED noise level σ, the denoising estimator recovers…

6.Is g(f(x)) − x guaranteed to be the gradient of some function for ANY encoder-decoder pair?

6 questions