§16.2.4–16.2.5Energy-Based Models · Separation and D-Separation

§16.2.4 Energy-based models

Many useful theoretical results about undirected models require that $\tilde{p} (\boldsymbol{x}) > 0$ everywhere. The cleanest way to guarantee it is to write the unnormalized probability as the exponential of a negated energy function energy function E(x): the exponent in an EBM, p̃(x)=exp(−E(x)); its additive terms correspond to cliques/factors, so an EBM is a product of experts. defined in ch. 16 — open in glossary — an energy-based model energy-based model An undirected model written p̃(x)=exp(−E(x)), guaranteeing p̃>0 and allowing unconstrained learning of the energy function E. defined in ch. 16 — open in glossary (EBM):

Learning an energy function is easier than learning clique potentials directly: because any $E$ already gives a positive $\tilde{p}$, training is unconstrained optimization rather than constrained (we never have to manually enforce a minimum probability). Any distribution of this exponential form is a Boltzmann distribution boltzmann distribution Any distribution of the form p̃(x)=exp(−E(x)); models of this form are often called Boltzmann machines. defined in ch. 16 — open in glossary , and for that reason many energy-based models are called Boltzmann machines boltzmann machine An energy-based model, today usually one WITH latent variables (an EBM without latents is more often called a Markov random field / log-linear model). defined in ch. 16 — open in glossary (Fahlman et al., 1983; Ackley et al., 1985; Hinton and Sejnowski, 1986). The terminology has drifted: the name once meant a purely binary model, but today real-valued variants exist (e.g. the mean–covariance RBM), and “Boltzmann machine” now usually designates a model with latent variables — without them, the same object is more often called a Markov random field or log-linear model.

Energy terms are cliques. Since $\exp(a)\exp(b) = \exp(a+b)$, a product of clique potentials becomes a sum of per-clique energies: each additive term of $E$ corresponds to one clique. Reading an energy off a graph is therefore just reading its cliques, with $\phi_{C} = \exp(-E_{C})$:

When a model has latent variables $\boldsymbol{h}$, some algorithms are phrased in terms of the free energy free energy F(x) = −log Σₕ exp(−E(x,h)): the energy of x after marginalizing the latent variables h in an EBM. defined in ch. 16 — open in glossary — the energy of $\boldsymbol{x}$ after marginalizing them out:

§16.2.5 Separation and d-separation

Edges tell us which variables interact directly. We usually also want to know which variables are conditionally independent — and crucially, some indirect interactions can be switched on or off by observing other variables.

In an undirected graph this is called separation separation Conditional independence implied by an UNDIRECTED graph: A⊥B|S iff every path from A to B passes through an observed (inactive) variable in S. defined in ch. 16 — open in glossary : a set $A$ is separated from $B$ given $S$ if every path between them passes through an observed variable. A path through only unobserved variables is an active path active path A path whose intermediate variables leave its endpoints dependent; blocked/'inactive' paths do not. Observing a variable can block a path — or, at a collider, unblock it. defined in ch. 16 — open in glossary (the endpoints stay dependent); a path containing an observed variable is inactive (blocked). We shade observed variables in.

For directed graphs the same idea is called d-separation d-separation The directed-graph analogue of separation ('d' = dependence); the active-path rules are more subtle (chain, common cause, collider, descendant). defined in ch. 16 — open in glossary (“d” for dependence), but deciding whether a path is active is subtler. Every length-2 path falls into one of four patterns — and the collider behaves backwards from the rest. Explore all four: click the middle variable $s$ (and, in the last case, the child $c$) to observe or un-observe it, and watch independence flip:

Reading these rules off a larger graph is mechanical. Take the book’s fig 16.9 (a and b both parents of c; c the parent of d and e) and click through it:

Two cautions close the section. A graph never implies an independence that is false — but it may fail to encode one that is true. When $a=0$ makes $b$ and $c$ independent while $a=1$ forces $b=c$, that context-specific independence context-specific independence An independence that holds only for certain values of some variable; it cannot be expressed by graph structure alone. defined in ch. 16 — open in glossary cannot be drawn as graph structure (the edge needed for $a=1$ hides the independence at $a=0$). And in the worst case the complete graph — every possible edge — represents any distribution at all, precisely because it implies no independences. The art is to imply as many true independences as possible, and no false ones.

Next: converting between the two graph languages, factor graphs, sampling, and why structure helps.