Semi-Supervised Disentangling of Causal Factors

§15.3 The ideal representation: causes, separated

Back to the chapter’s central question — what makes one representation better than another? One answer: an ideal representation’s features correspond to the underlying causes of the data, with separate features or directions for separate causes — a representation that disentangles the causes from one another. This hypothesis motivates the whole seek-a-representation-of- $p(\boldsymbol{x})$ -first program: if $y$ is among the most salient causes of $\boldsymbol{x}$ , that representation should serve $p(y \mid \boldsymbol{x})$ too — an idea guiding research since at least the 1990s (Becker & Hinton 1992; Hinton & Sejnowski 1999).

Note the tension with earlier chapters: we have often wanted representations that are easy to model — sparse (§13.4), independent (§13.2). A causally-disentangled representation is not automatically easy to model. The deeper part of the hypothesis is that for many AI tasks the two properties coincide: once you hold the underlying explanation of the data, isolating individual attributes becomes easy.

When does knowing $p(\boldsymbol{x})$ help at all? It can fail: if $p(\boldsymbol{x})$ is uniform, a training set of $\boldsymbol{x}$ values carries zero information about $\mathbb{E}[y \mid \boldsymbol{x}]$ — no amount of unsupervised learning helps. And it can succeed spectacularly: when $\boldsymbol{x}$ arises from a mixture with one component per class (fig 15.4), modeling $p(\boldsymbol{x})$ alone finds the components, and one label per class finishes the job. Try both regimes:

Fig 15.4, interactive (reused mixture explorer) — a density over x that is a mixture. Drag the component means far apart: p(x) alone reveals where each class lives, and a single label per component would pin down p(y|x) almost perfectly. Slide them together: the components blur into one bump, and unsupervised modeling of p(x) stops revealing the class factor.

α0.50μ₁-1.20μ₂1.20σ0.50

P(x) = α·N(x; μ₁, σ²) + (1−α)·N(x; μ₂, σ²): the prior α = P(c=1) = 0.50 picks the component; merge the means and the modes fuse

What ties the marginal to the conditional in general? Causality. Suppose $y$ is one of the causal factors of $\boldsymbol{x}$ , and let $\boldsymbol{h}$ stand for all of them:

The causal structure, and the Bayes tie (eqs 15.1–15.3)

p(\boldsymbol{h}, \boldsymbol{x}) = p(\boldsymbol{x} \mid \boldsymbol{h})\, p(\boldsymbol{h}), \qquad p(\boldsymbol{x}) = \mathbb{E}_{\boldsymbol{h}}\, p(\boldsymbol{x} \mid \boldsymbol{h}), \qquad p(y \mid \boldsymbol{x}) = \frac{p(\boldsymbol{x} \mid y)\, p(y)}{p(\boldsymbol{x})}

$p(\boldsymbol{h}, \boldsymbol{x}) = p(\boldsymbol{x} \mid \boldsymbol{h}) p(\boldsymbol{h})$	the true generative process: h (all causal factors) is the parent of x in the directed graphical model — compare fig 13.1	eq 15.1
$p(\boldsymbol{x}) = \mathbb{E}_{\boldsymbol{h}} p(\boldsymbol{x} \mid \boldsymbol{h})$	the marginal the unsupervised learner sees — the best model of x (for generalization) is the one that uncovers the TRUE latent structure	eq 15.2
$p(y \mid \boldsymbol{x}) = p(\boldsymbol{x} \mid y) p(y) / p(\boldsymbol{x})$	Bayes ties the conditional to the same components — knowledge of p(x)'s structure is knowledge ABOUT p(y\|x), so semi-supervised learning should help whenever these assumptions hold	eq 15.3

Worked example — the Bayes tie with numbers

Two classes, equal priors $p(y{=}1) = p(y{=}2) = 0.5$ , with $p(x \mid y{=}1) = \mathcal{N}(x; -2, 1)$ and $p(x \mid y{=}2) = \mathcal{N}(x; 2, 1)$ — a mixture the unsupervised learner can discover from unlabeled $x$ alone. At $x = -1.5$ : the density ratio is $e^{-0.5(0.5)^2} / e^{-0.5(3.5)^2} = e^{5.88} \approx 358$ , so $p(y{=}1 \mid x{=}-1.5) \approx 358/359 \approx 99.7\%$ . The posterior was sitting inside the mixture structure all along — the labels only needed to say which bump is which.

Too many causes — and who decides what is salient?

Real observations have an enormous number of underlying causes. If $y = h_i$ but we don’t know which $i$ , the brute-force plan is to capture all reasonably salient factors and disentangle them — then predicting $y$ is easy regardless. In practice that is infeasible: should a visual representation encode every small background object? (Even humans don’t — we famously fail to perceive changes irrelevant to our current task; Simons & Levin 1998.) Determining what to encode is a research frontier, with two main working strategies: use a supervised signal simultaneously with the unsupervised one so the model favors task-relevant factors, or use much larger representations when purely unsupervised.

An emerging third strategy changes the definition of salience itself. Fixed criteria decide which causes count: mean squared error implicitly says a cause is salient only if it moves the brightness of many pixels — which is how an autoencoder for a robotics task can fail to reconstruct a ping-pong ball whose existence and position are exactly what the task needs (fig 15.5). A pattern can instead be salient because it is highly recognizable:

Figs 15.5–15.6, schematically — salience is chosen by the loss. Left: under MSE, a large object survives reconstruction while a small-but-crucial ball is dropped (too few pixels). Right: a predictive model of heads trained with MSE alone omits the ears (no extreme brightness); adding an ADVERSARIAL loss restores them — a discriminator recognizes their consistent pattern, making them salient (Lotter et al. 2015). Generative adversarial networks (§20.10.4) learn what is salient.

The causal dividend: robustness for free

Schölkopf et al. (2012) point out a benefit of getting the causal direction right. If $\boldsymbol{x}$ is the effect and $y$ the cause, then the mechanism $p(\boldsymbol{x} \mid y)$ is robust to changes in $p(y)$ — modeled the other way around, Bayes’ rule makes the conditional sensitive to those shifts. And distribution changes in the wild — new domains, temporal non-stationarity, new tasks — mostly change the marginal over causes while the causal mechanisms stay invariant (the laws of the universe are constant). A generative model that recovers $\boldsymbol{h}$ and $p(\boldsymbol{x} \mid \boldsymbol{h})$ inherits that invariance.

Next: the tool that makes cause-per-direction representations possible at all — distributed representations, and the exponential gains of depth.

Check yourself — disentangling causal factors

1.What is the chapter's "ideal representation" hypothesis?

2.When is unsupervised learning of p(x) PROVABLY useless for p(y|x)?

3.Predict the outcome — in the mixture explorer, slide the two component means far apart. Why does this make semi-supervised learning succeed?

4.The robot autoencoder failed to reconstruct a ping-pong ball (fig 15.5). What went wrong, and what fixes this class of failure?

5.Per Schölkopf et al. (2012), why prefer modeling in the CAUSAL direction (x as effect, y as cause)?

6.Capturing ALL underlying causes is infeasible (a scene has countless tiny background factors). What are the two main practical strategies?

6 questions