Pseudolikelihood · Score Matching · Denoising Score Matching · Noise-Contrastive Estimation

The MCMC methods of §18.2 confront the partition function directly. The methods here sidestep it — they train a model without ever computing $Z$ . Almost all rest on one observation: in a ratio of probabilities, $Z$ cancels.

Ratios cancel the partition function (eq 18.17)

\frac{p(\boldsymbol{x})}{p(\boldsymbol{y})} = \frac{\frac{1}{Z}\tilde{p}(\boldsymbol{x})}{\frac{1}{Z}\tilde{p}(\boldsymbol{y})} = \frac{\tilde{p}(\boldsymbol{x})}{\tilde{p}(\boldsymbol{y})}

$\frac{1}{Z}$	the same intractable normalizer sits in numerator and denominator	scalar
$\frac{\tilde{p}(\boldsymbol{x})}{\tilde{p}(\boldsymbol{y})}$	so it cancels — a ratio of UNNORMALIZED probabilities is Z-free and cheap	scalar

§18.3 Pseudolikelihood

A conditional distribution is a ratio, so it needs no $Z$ . The pseudolikelihood replaces the true joint likelihood with a product of one-variable conditionals:

The pseudolikelihood objective (eq 18.20)

\sum_{i=1}^{n} \log p(\mathrm{x}_i \mid \boldsymbol{x}_{-i})

$p(\mathrm{x}_i \mid \boldsymbol{x}_{-i})$	predict each variable from ALL the others — a Z-free ratio (marginalize only the single variable xᵢ)	conditional
$\sum_{i=1}^{n}$	sum over all n variables. Cost: k·n evaluations of p̃ (k values × n variables) instead of the kⁿ needed for Z	n terms

This is not a hack: maximizing pseudolikelihood is asymptotically consistent. The generalized pseudolikelihood (eq 18.21) conditions on all-but-a-set $S^{(i)}$ , interpolating between pseudolikelihood (singletons) and the full log-likelihood (one set of everything) — useful when structure lets you group correlated variables (e.g. local windows in images). Pseudolikelihood does poorly on tasks needing the full joint (density estimation, sampling) but can beat maximum likelihood on tasks that only use the conditionals it trains (filling in a few missing values). Its Achilles’ heel: $\tilde{p}$ sits in a denominator, so a lower bound on $\tilde{p}$ (as from variational inference, chapter 19) is useless — it only upper-bounds the objective.

§18.4 Score matching and ratio matching

Score matching avoids $Z$ a different way. The score is the gradient of the log-density with respect to the input, $\nabla_{\boldsymbol{x}} \log p(\boldsymbol{x})$ — and since $Z$ does not depend on $\boldsymbol{x}$ , $\nabla_{\boldsymbol{x}} \log Z = 0$ : the score is Z-free.

The score-matching objective (eq 18.22)

L(\boldsymbol{x}, \boldsymbol{\theta}) = \tfrac{1}{2}\left\lVert \nabla_{\boldsymbol{x}} \log p_\text{model}(\boldsymbol{x}) - \nabla_{\boldsymbol{x}} \log p_\text{data}(\boldsymbol{x}) \right\rVert_2^2

$\nabla_{\boldsymbol{x}} \log p_\text{model}$	the model score — computable from p̃ alone, because ∇_x log Z = 0	vector (dim n)
$\nabla_{\boldsymbol{x}} \log p_\text{data}$	the data score — seemingly needs the true p_data, but the expected loss rewrites (eq 18.25) into 2nd derivatives of log p_model only	vector (dim n)
$\tfrac{1}{2}\lVert \cdot \rVert_2^2$	match the two score fields in squared error; minimized when the model's log-density has the same slope everywhere as the data's	scalar

Matching the slope of the log-density everywhere is matching a vector field — the same field the denoising autoencoder learns (§14.5.1). Revisit it:

The score field ∇_x log p(x), recalled from §14.5.1. Score matching trains a model so this field matches the data's — arrows everywhere point "uphill" toward the data manifold. Because the field is a derivative w.r.t. x, the partition function (constant in x) never enters.

corruption width σ = 26

Each arrow is the optimal denoising reconstruction minus the input, g(f(x̃)) − x̃ = 𝔼[x | x̃] − x̃ — the center of mass of the clean points that could have produced x̃, minus x̃. It estimates the score ∇ₓ log p(x) (up to a factor σ²): arrows climb toward the manifold and shrink to dots where the estimated density is flat or maximal. Moderate σ: a smooth field pointing cleanly back to the spiral — zeros ON the manifold (density maxima) and between the arms (density minima), long arrows in low-probability gaps where a step uphill gains the most.

Score matching needs first and second $\boldsymbol{x}$ -derivatives, so it applies to continuous data only and, like pseudolikelihood, is incompatible with lower bounds on $\log\tilde{p}$ . For binary/discrete data the analogue is ratio matching (eq 18.26): it compares each data point to its single-bit flips, again cancelling $Z$ in the ratio, and pushes down every fantasy state within Hamming distance 1 of the data.

§18.5 Denoising score matching

A consistent estimator with enough capacity collapses $p_\text{model}$ onto Dirac spikes at the training points. Denoising score matching regularizes this by score-matching a noise-smoothed target instead of the raw empirical distribution:

Smoothing the target (eq 18.27)

p_\text{smoothed}(\boldsymbol{x}) = \int p_\text{data}(\boldsymbol{y})\, q(\boldsymbol{x} \mid \boldsymbol{y})\, d\boldsymbol{y}

$q(\boldsymbol{x} \mid \boldsymbol{y})$	a corruption process — usually add a little noise to each data point y	noise kernel
$p_\text{smoothed}$	the blurred data distribution: no longer spikes, so the fitted model generalizes (at the cost of the exact-consistency property of §5.4.5)	density

This is exactly why several autoencoder objectives (§14.5.1) turn out to be score matching in disguise — training a denoising autoencoder is a way around the partition function.

§18.6 Noise-contrastive estimation

Noise-contrastive estimation (NCE) makes the normalizer an explicit learned parameter: $\log p_\text{model}(\boldsymbol{x}) = \log \tilde{p}(\boldsymbol{x}) + c$ , with $c$ estimated alongside $\boldsymbol{\theta}$ as an approximation of $-\log Z$ (maximum likelihood couldn’t do this — it would drive $c \to \infty$ ). The trick: reduce density estimation to binary classification of data versus a tractable noise distribution $p_\text{noise}$ . The optimal classifier turns out to be a logistic regression on the log-density difference:

The NCE classifier is a sigmoid of the log-ratio (eqs 18.33–18.37)

p_\text{joint}(y=1 \mid \boldsymbol{x}) = \frac{p_\text{model}(\boldsymbol{x})}{p_\text{model}(\boldsymbol{x}) + p_\text{noise}(\boldsymbol{x})}

step 1/4: Bayes with equal priors p(y=1)=p(y=0)=½: the posterior that x came from the model, not the noise.

Fit that classifier by ordinary maximum likelihood and you get a consistent estimator of $p_\text{model}$ . Tune the model and watch the classifier — the objective is best exactly when the model matches the data:

NCE as classification. σ(log p_model − log p_noise) is the probability the classifier assigns to "data, not noise". Slide p_model (μ, width): the classification objective J peaks precisely when p_model = p_data — density estimation solved by classification. The learned constant c absorbs −log Z, so p_model need not be normalized.

μ = -1.20width = 1.30

classifier objective J = -1.722

best possible = -1.056 (gap 0.666)

NCE trains p_model by making a binary classifier — σ(log p_model − log p_noise) — best separate data from noise. That objective is maximized exactly when p_model matches p_data, so a density-estimation problem becomes a classification problem. (A learned constant c absorbs −log Z, so p_model needn't be normalized.)

NCE shines when there are few variables (even with many values — e.g. a word given its context) but slows on high-dimensional data: the classifier can reject a noise sample by spotting any single wrong variable, so learning stalls once the marginals are right. When the noise distribution is set to the current model before each step, NCE becomes self-contrastive estimation , whose expected gradient equals the maximum-likelihood gradient. And the idea that “a good generative model should fool a classifier” is one step from generative adversarial networks (§20.10.4).

The Z-confronting toolbox at a glance

The methods of this chapter compared. Click a cell for detail.
	Method	Handles Z by	Discrete data?	Lower-bound p̃ ok?
CD / SML	CD / SML (§18.2)	estimates ∇log Z via model samples	yes	yes
Pseudolikelihood	pseudolikelihood	ratios cancel Z	yes	no
Score matching	score matching	∇_x log Z = 0	no (continuous only)	no
Ratio matching	ratio matching	bit-flip ratios cancel Z	yes (binary)	no
NCE	noise-contrastive estimation	learns c ≈ −log Z	yes	no

Dotted-underlined cells have explanations — click one.

Check yourself — pseudolikelihood, score matching & NCE

1.What single observation underlies most methods that train undirected models without computing Z?

2.How does pseudolikelihood reduce cost, and what is its main weakness?

3.Why does score matching avoid the partition function, and what does that restrict?

4.In NCE, the optimal data-vs-noise classifier simplifies to which form?

5.Predict the outcome — in NCELab you slide p_model's mean and width to maximize the classifier objective J. Where is J largest?

5 questions