Structured Probabilistic Models · Deep Learning Interactive

Machine learning routinely needs distributions over very many random variables — but usually the direct interactions are between few of them. Representing the entire joint with a single function is inefficient computationally and statistically. The alternative: split the distribution into factors to multiply together. If $\mathrm{a}$ influences $\mathrm{b}$ , $\mathrm{b}$ influences $\mathrm{c}$ , and $\mathrm{a} \bot \mathrm{c} \mid \mathrm{b}$ , then $p(\mathrm{a}, \mathrm{b}, \mathrm{c}) = p(\mathrm{a})\, p(\mathrm{b}\mid\mathrm{a})\, p(\mathrm{c}\mid\mathrm{b})$ — each factor over fewer variables.

Worked example — why factoring wins

Factors cost parameters exponential in the variables they touch. Five binary variables as one raw joint: $2^5 - 1 = 31$ numbers. With the factorization of fig 3.7 below: $p(a){:}1 + p(b|a){:}2 + p(c|a{,}b){:}4 + p(d|b){:}2 + p(e|c){:}2 = \mathbf{11}$ . With 10-state variables the gap explodes: $10^5{-}1 = 99{,}999$ vs $9 + 90 + 900 + 90 + 90 = 1{,}179$ .

We describe such factorizations with graphs (in the graph-theory sense) — hence structured probabilistic models , or graphical models: each node is a random variable; an edge means the distribution can represent a direct interaction between the two.

Directed models

Directed models factor into conditional distributions: one factor per variable, conditioned on its parents in the graph:

Term by term

p(\boldsymbol{\mathrm{x}}) = \prod_i p\big(\mathrm{x}_i \mid Pa_{\mathcal{G}}(\mathrm{x}_i)\big)

$Pa_{\mathcal{G}}(\mathrm{x}_i)$	the parents of xᵢ — the nodes with arrows INTO it	node set
$p(\mathrm{x}_i \mid \cdot)$	each factor is a genuine conditional distribution — normalized on its own	one per variable

Fig 3.7 (interactive) — a directed model; click nodes

p(\mathrm{a},\mathrm{b},\mathrm{c},\mathrm{d},\mathrm{e}) =

p(\mathrm{a})

p(\mathrm{b} \mid \mathrm{a})

p(\mathrm{c} \mid \mathrm{a}, \mathrm{b})

p(\mathrm{d} \mid \mathrm{b})

p(\mathrm{e} \mid \mathrm{c})

click a node to see which factors mention it

Reading the graph is fast: $\mathrm{a}$ and $\mathrm{c}$ interact directly; $\mathrm{a}$ and $\mathrm{e}$ only indirectly, via $\mathrm{c}$ .

Undirected models

Undirected models factor into plain functions — not distributions. Any fully-connected set of nodes is a clique $\mathcal{C}^{(i)}$ , and each clique gets a factor $\phi^{(i)}(\mathcal{C}^{(i)}) \ge 0$ . Bigger factor value = more likely configuration — but nothing makes the product sum to 1, so we divide by the normalizing constant $Z$ :

Term by term

p(\boldsymbol{\mathrm{x}}) = \frac{1}{Z} \prod_i \phi^{(i)}\big(\mathcal{C}^{(i)}\big)

$\phi^{(i)}$	a clique factor: non-negative, but NOT normalized — just a compatibility score	function ≥ 0
$\mathcal{C}^{(i)}$	a clique — a set of mutually connected nodes	node set
$Z$	the sum/integral of the product over ALL states — the price of using unnormalized factors (ch. 18 is devoted to this object)	scalar

Fig 3.8 (interactive) — the undirected version; click nodes

p(\mathrm{a},\mathrm{b},\mathrm{c},\mathrm{d},\mathrm{e}) = \tfrac{1}{Z}\,

\phi^{(1)}(\mathrm{a}, \mathrm{b}, \mathrm{c})

\phi^{(2)}(\mathrm{b}, \mathrm{d})

\phi^{(3)}(\mathrm{c}, \mathrm{e})

click a node to see which factors mention it

What this really says

Directed vs undirected is a property of the description, not of the distribution: any distribution can be written both ways. The graphs are a language for saying which direct relationships a model chooses to represent — not mutually exclusive families. Parts I–II use this language only descriptively; Part III (ch. 16 onward) builds the machinery.

That closes the probability toolkit. One more set of fundamentals remains before machine learning proper: numerical computation — chapter 4.

Check yourself — structured probabilistic models

1.Five binary variables, factored per fig 3.7 vs one raw joint table. How do the parameter counts compare?

2.In the fig 3.7 widget you click node c. Predict what highlights.

3.What distinguishes undirected clique factors φ from the directed model's factors?

4.Is 'being directed' a property of a probability distribution?

4 questions