§13.0–13.3Linear Factor Models · Probabilistic PCA · ICA · Slow Feature Analysis

§13.0 Part III begins: probabilistic models with latent variables

Much of the research frontier of deep learning is about building a probabilistic model of the input, $p_{\text{model}}(\boldsymbol{x})$. Such a model can, in principle, use probabilistic inference to predict any variable in its environment given any of the others. Most of these models carry latent variables latent variable A random variable that is never observed directly, like the mixture's component identity c. defined in ch. 3 — open in glossary $\boldsymbol{h}$, with $p_{\text{model}}(\boldsymbol{x}) = \mathbb{E}_{\boldsymbol{h}}\, p_{\text{model}}(\boldsymbol{x} \mid \boldsymbol{h})$ — the latents provide a distributed representation of the data, with all the advantages we saw representation learning bring to feedforward and recurrent nets.

This chapter starts Part III with the simplest such models: linear factor models. They were among the first latent-variable models to be studied extensively, they still serve as building blocks of mixture models (Hinton et al. 1995; Ghahramani & Hinton 1996; Roweis et al. 2002) and of larger deep probabilistic models (Tang et al. 2012), and they exhibit — in miniature — most of the design questions the advanced generative models of chapters 16–20 will face.

A linear factor model linear factor model A simple latent-variable probabilistic model with a stochastic LINEAR decoder: sample factors h from a factorial prior, then generate x = Wh + b + noise. Probabilistic PCA, factor analysis, ICA, and sparse coding differ only in the noise and prior choices. defined in ch. 13 — open in glossary is defined by a stochastic, linear decoder: generate $\boldsymbol{x}$ by adding noise to a linear transformation of $\boldsymbol{h}$. The data-generation recipe is:

The corresponding directed graphical model (§3.14) is the shared family portrait — every $h_i$ can influence every $x_j$, and the observed variables are conditionally independent given the factors:

§13.1 Probabilistic PCA and factor analysis

Probabilistic PCA probabilistic pca A linear factor model with prior h~N(0,I) and ISOTROPIC Gaussian noise, giving x~N(b, WWᵀ+σ²I). Fit by EM; as σ→0 it becomes ordinary PCA (the posterior mean of h is the orthogonal projection onto Ws columns). defined in ch. 13 — open in glossary , factor analysis and other linear factor models are all special cases of eqs 13.1–13.2 that differ only in the noise distribution and the prior over $\boldsymbol{h}$.

In factor analysis (Bartholomew 1987; Basilevsky 1994) the prior is the unit Gaussian, and each observed $x_i$ gets its own noise variance:

To cast PCA in this probabilistic framework, make one small change: force all the conditional variances to be equal, a single scalar $\sigma^2$:

But the limit has a sting. As $\sigma \to 0$ the density concentrates ever more sharply around the $d$-dimensional subspace spanned by $\boldsymbol{W}$‘s columns — and any data point not near that hyperplane is assigned vanishingly low likelihood:

§13.2 Independent component analysis (ICA)

Independent component analysis independent component analysis (ica) A linear factor model that separates an observed signal into fully INDEPENDENT (not just decorrelated) underlying signals — e.g. the cocktail-party problem or removing heart/eye artifacts from EEG. Requires a non-Gaussian prior p(h) for W to be identifiable; typically learns sparse features. defined in ch. 13 — open in glossary is among the oldest representation-learning algorithms (Hérault & Ans 1984; Jutten & Hérault 1991; Comon 1994; Hyvärinen 1999). Its goal: separate an observed signal into underlying signals that were scaled and added together — and make the recovered signals fully independent, not merely decorrelated (the crucial distinction of §3.8).

The variant closest to our generative framing (Pham et al. 1992) is fully parametric: the user fixes the prior $p(\boldsymbol{h})$ ahead of time, the model generates $\boldsymbol{x} = \boldsymbol{W}\boldsymbol{h}$ deterministically (no added noise), and a nonlinear change of variables (eq 3.47) yields $p(\boldsymbol{x})$ for maximum-likelihood training. Choosing an independent $p(\boldsymbol{h})$ is the lever: it forces the recovered factors to be as close to independent as possible.

This is used less for abstract causal factors than for low-level signal separation. Each training example is one moment in time; each $x_i$ is one sensor’s reading of the mixed signals; each $h_i$ is one recovered source. The classic cocktail-party problem: $n$ people speak simultaneously into $n$ microphones placed at different spots — ICA separates the voices so each $h_i$ contains one speaker, clearly. In neuroscience, EEG electrodes on the scalp record brain activity mixed with far stronger heartbeat and eye-blink artifacts; ICA separates the heart’s electrical signature from the brain’s, and brain regions from one another. Open the box and do the separation yourself:

Beyond this variant, “ICA” names a whole family. Some members add generation noise; most drop maximum likelihood and instead directly push $\boldsymbol{h} = \boldsymbol{W}^{-1}\boldsymbol{x}$ toward independence. Since eq 3.47 needs $\det(\boldsymbol{W})$ — expensive and numerically unstable — many variants constrain $\boldsymbol{W}$ to be orthogonal. Two facts organize the family:

• All variants require a non-Gaussian $p(\boldsymbol{h})$. With an independent Gaussian prior, $\boldsymbol{W}$ is not identifiable — many different $\boldsymbol{W}$ produce the same $p(\boldsymbol{x})$, because a rotated Gaussian is still the same Gaussian. This is the opposite of probabilistic PCA and factor analysis, which want Gaussian priors for their closed forms. In the maximum-likelihood approach the user typically picks $p(h_i) = \frac{d}{dh_i}\sigma(h_i)$ — and such choices put larger peaks near 0 than a Gaussian, so most ICA implementations effectively learn sparse features.
• Many variants are not generative models in our sense: they can transform between $\boldsymbol{x}$ and $\boldsymbol{h}$ but represent neither $p(\boldsymbol{h})$ nor $p(\boldsymbol{x})$. For instance, maximizing the sample kurtosis of $\boldsymbol{h} = \boldsymbol{W}^{-1}\boldsymbol{x}$ drives non-Gaussianity (as in the widget) without ever writing down a density — fine for ICA’s usual role as an analysis tool rather than a sampler.

Generalizations. Just as PCA generalizes to the nonlinear autoencoders of chapter 14, ICA generalizes to nonlinear ICA nonlinear ica A generative generalization of ICA using a nonlinear encoder/decoder; NICE (Dinh 2014) stacks invertible transforms whose Jacobian determinants are cheap, enabling exact likelihood and easy sampling. defined in ch. 13 — open in glossary (Hyvärinen & Pajunen 1999), with successful ensemble-learning applications (Roberts & Everson 2001; Lappalainen et al. 2000). NICE (Dinh et al. 2014) stacks invertible encoder stages whose Jacobian determinants are cheap to compute, giving exact likelihood — and since the decoder is the encoder’s perfect inverse, sampling is as easy as drawing $\boldsymbol{h}$ and decoding. Relaxing full independence instead: independent subspace analysis independent subspace analysis An ICA generalization that learns non-overlapping GROUPS of features, allowing statistical dependence within a group but discouraging it between groups. defined in ch. 13 — open in glossary allows dependence within non-overlapping feature groups while discouraging it between groups; and topographic ICA topographic ica An ICA variant giving hidden units spatial coordinates and forming overlapping groups of neighbors, so nearby features share orientation/location/frequency; on images it learns Gabor filters and pooling gives translation invariance. defined in ch. 13 — open in glossary gives each unit a spatial coordinate with overlapping neighbor groups, so nearby units learn similar features. Trained on natural images, it learns Gabor filters gabor function A Gaussian gate × oriented cosine (eq 9.16) describing V1 simple-cell weights: it responds to a specific spatial frequency of brightness in a specific direction at a specific location; countless learning algorithms learn Gabor-like first-layer filters. defined in ch. 9 — open in glossary whose neighbors share orientation, location or frequency — and pooling over each small region yields translation invariance, echoing the complex-cell story of §9.10.

§13.3 Slow feature analysis

Slow feature analysis slow feature analysis (sfa) A linear factor model applying the slowness principle in closed form: learn features that vary as slowly as possible over a time sequence, subject to zero mean, unit variance, and mutual decorrelation. Features come ordered slowest-first; nonlinear via a basis expansion. defined in ch. 13 — open in glossary (Wiskott & Sejnowski 2002) is a linear factor model that uses information from time signals to learn invariant features. It rests on the slowness principle slowness principle The idea that important characteristics of a scene change SLOWLY compared with raw measurements (a zebras stripes flicker a pixel fast, but zebra present does not) — so regularize features to change slowly over time (penalty λΣₜ L(f(x⁽ᵗ⁺¹⁾),f(x⁽ᵗ⁾))). defined in ch. 13 — open in glossary : the important characteristics of a scene change slowly compared with the raw measurements describing it. Watch a zebra cross an image: any individual pixel flips black ↔ white rapidly as stripes pass over it, the feature “the zebra’s position” changes slowly, and “is a zebra present” does not change at all. So: regularize the model to learn features that change slowly.

The principle predates SFA (Hinton 1989; Földiák 1989; Mobahi et al. 2009; Bergstra & Bengio 2009) and can be added to any differentiable model trained by gradient descent, as a cost term:

SFA is the especially efficient special case: apply the principle to a linear feature extractor $f(\boldsymbol{x}; \boldsymbol{\theta})$, and the optimum has a closed-form solution — no gradient descent at all. (Like the non-generative ICA variants, SFA defines a map from input to feature space but no prior over features, hence no $p(\boldsymbol{x})$.) The problem:

Deep and nonlinear SFA. SFA learns nonlinear features by applying a nonlinear basis expansion first — commonly quadratic, replacing $\boldsymbol{x}$ with the vector of all products $x_i x_j$ — and then running linear SFA. Stacking (linear SFA → expansion → linear SFA → …) yields deep slow feature extractors. Trained on natural video patches, quadratic SFA learns features with the hallmarks of complex cells complex cell A V1 cell responding like a simple cell but invariant to small position shifts — the inspiration for pooling units; ≈ the L² norm of a quadrature pair of simple cells. defined in ch. 9 — open in glossary in V1 (Berkes & Wiskott 2005); trained on random 3-D camera motion, deep SFA learns features resembling the navigation cells of rat brains (Franzius et al. 2007) — a reasonably biologically plausible model.

A distinctive advantage: SFA’s output is theoretically predictable even in the deep nonlinear setting. Knowing how the environment’s underlying factors change (e.g. the distribution over camera position and velocity), one can solve analytically for the optimal features — and deep SFA experiments do recover the predicted functions, where most learning algorithms’ outcomes depend inscrutably on pixel-level details of the cost. It has also been used for object recognition and pose estimation (Franzius et al. 2008). Yet the slowness principle has powered no state-of-the-art application so far. One speculation: the prior is too strong — demanding features be approximately constant punishes the position of a fast-moving object, when what we really want is features that are easy to predict from one step to the next. An object’s position is useful whether its velocity is high or low.

Next: a linear factor model whose encoder is an optimization problem — sparse coding, and the manifold view of PCA.