RNNs as Directed Graphical Models · Conditioning on Context

§10.2.3 Recurrent networks as directed graphical models

So far our loss $L^{(t)}$ has been the cross-entropy between target $y^{(t)}$ and output $o^{(t)}$ . As with a feedforward net, almost any loss works — but we usually interpret the output as a probability distribution and use its cross-entropy. (Mean squared error, for instance, is just the cross-entropy of a unit-Gaussian output.) Training with a predictive log-likelihood objective trains the RNN to estimate the conditional distribution of the next element given the past — either $\log p(y^{(t)} \mid x^{(1)}, \dots, x^{(t)})$ , or, if outputs feed back as inputs, $\log p(y^{(t)} \mid x^{(1)}, \dots, x^{(t)}, y^{(1)}, \dots, y^{(t-1)})$ .

Take the purest case: an RNN modeling a sequence of scalars $y$ with no inputs $x$ , where the input at step $t$ is simply the output at $t-1$ . By the chain rule of probability, the joint factorizes as a product of one-step predictions — and every past value may condition the present:

The sequence joint, factorized (eqs 10.31–10.33)

P(\mathbf{y}) = \prod_{t=1}^{\tau} P\!\big(y^{(t)} \mid y^{(t-1)}, y^{(t-2)}, \dots, y^{(1)}\big), \qquad L = \sum_t -\log P\!\big(y^{(t)} \mid y^{(t-1)}, \dots, y^{(1)}\big)

$\prod_{t=1}^{\tau}$	chain rule (eq 3.6): the joint over the whole sequence is a product of one-step conditional predictions	τ factors
$P(y^{(t)} \mid y^{(t-1)},\dots,y^{(1)})$	each step conditions on the ENTIRE past — no Markov cutoff	distribution
$L = \sum_t -\log P(\cdots)$	the negative log-likelihood is again a sum of per-step terms	scalar

Drawn as a graphical model over the $y$ values alone (marginalizing out the hidden units), this is the complete graph (fig 10.7): every past $y^{(i)}$ links directly to each future $y^{(t)}$ — an ever-growing fan-in. But re-introduce the hidden state $h^{(t)}$ as a node (fig 10.8) and something remarkable appears: $h$ decouples the past from the future, every stage becomes identical, and the parameters can be shared. Toggle the two views, then grow $\tau$ and the vocabulary $k$ to watch the tabular cost explode while the RNN’s stays flat:

Figs 10.7–10.8 (interactive) — the sequence as a directed graphical model. Complete graph: every past y links to each future y. Hidden-state view: h decouples past and future, so one shared f gives full connectivity at O(1) parameters — versus the tabular O(kᵗ).

length τ = 5vocab k = 3

Tabular joint P(y⁽¹⁾,…,y⁽τ⁾)

O(kᵗ) = 3⁵ = 243 entries

grows exponentially with sequence length

RNN (shared f, θ)

O(1) in τ — same parameter count

one transition function reused at every step

Complete graph: every past yⁱ may directly influence a future yᵗ — an ever-growing fan-in and, tabulated, an O(kᵗ) blow-up.

Full connectivity, efficient parametrization

Represent an arbitrary joint over τ discrete $k$ -ary variables as a lookup table and you need $O(k^\tau)$ entries — hopeless for long sequences. The RNN obtains the same full connectivity (any past value can influence any future one, via $h$ ) with $O(1)$ parameters in $\tau$ , by reusing the same function $f$ and parameters $\theta$ at every step. The graphical-model view makes the trade explicit — and its price: fewer parameters can make optimization harder, and some operations (like predicting a missing value in the middle of a sequence) stay difficult.

Why sharing is allowed — stationarity. Using the same parameters at every step is equivalent to assuming the conditional distribution at step $t+1$ given step $t$ is stationary : the relationship between consecutive steps does not depend on $t$ . (One could feed $t$ as an extra input to let the model discover time-dependence while still sharing — better than a separate distribution per $t$ , but then the network must extrapolate to new values of $t$ .)

Sampling — how does generation stop? To sample from the model we draw from the conditional at each step, but we also need a mechanism to decide the sequence length $\tau$ . Three options:

Determining sequence length when sampling — click a cell for detail.
	How it works	Applies to
End-of-sequence symbol	Add a special ⟨EOS⟩ token; stop when it is sampled.	symbol-vocabulary outputs only
Bernoulli halt output	An extra sigmoid output each step decides continue vs stop.	any RNN
Predict τ directly	An extra output predicts the integer length τ; feed τ−t as an extra input each step.	any RNN

Dotted-underlined cells have explanations — click one.

§10.2.4 Modeling sequences conditioned on context

An RNN need not model only $P(\mathbf{y})$ — it can model $P(\mathbf{y} \mid \mathbf{x})$ . As with feedforward nets (§6.2.1), any model of $P(\mathbf{y}; \theta)$ becomes a model of $P(\mathbf{y} \mid \omega)$ by making the parameters $\omega$ a function of the input $\mathbf{x}$ .

A single fixed vector as context. When $\mathbf{x}$ is one fixed-size vector (say, an image), it can enter the $y$ -generating RNN as (1) an extra input at every step, (2) the initial state $h^{(0)}$ , or (3) both. The most common choice (fig 10.9) introduces a new weight matrix $R$ ; the same product $\mathbf{x}R$ is added to the hidden units at every step — effectively an input-dependent bias. This is exactly the shape of an image-captioning model: one image in, a sentence out.

Fig 10.9 — a fixed vector x mapped to a distribution over sequences. The new matrix R injects x·R into every hidden state (an input-dependent bias). Used for image captioning: one image → a sequence of words.

A sequence as context. When the input is itself a sequence $x^{(t)}$ , the RNN of eq 10.8 corresponds to a conditional distribution that makes a conditional independence assumption:

The conditional-independence assumption (eq 10.35)

P\big(y^{(1)}, \dots, y^{(\tau)} \mid x^{(1)}, \dots, x^{(\tau)}\big) = \prod_{t} P\big(y^{(t)} \mid x^{(1)}, \dots, x^{(t)}\big)

$\prod_t$	the joint over outputs factorizes into independent per-step predictions — the y values are conditionally independent GIVEN the x sequence	τ factors
$P(y^{(t)} \mid x^{(1)},\dots,x^{(t)})$	each y⁽ᵗ⁾ depends on the inputs up to t, but NOT directly on the other y values	distribution

To remove that assumption — to let the $y$ values depend on each other — add connections from the output at step $t$ to the hidden unit at step $t+1$ (fig 10.10, another use of a matrix $R$ ). Now the model can represent an arbitrary distribution over the $y$ sequence:

Fig 10.10 — a conditional RNN with output→hidden feedback (R). Unlike fig 10.3, the previous output feeds the next hidden state, so the y values are no longer conditionally independent given x. One restriction remains: input and output sequences share the same length (lifted in §10.4).

That output-to-hidden model still has one restriction — input and output sequences must be the same length. Lifting that gives the encoder–decoder architecture: bidirectional RNNs, deep & recursive nets, and sequence-to-sequence.

Check yourself — RNNs as graphical models and conditioning on context

1.Why is an RNN's parametrization of the joint distribution over a sequence so much more efficient than a tabular representation?

2.In the hidden-state graphical-model view (fig 10.8), what role does h⁽ᵗ⁾ play?

3.Parameter sharing across time relies on which assumption?

4.When sampling from an RNN, how can the model decide when to STOP generating (determine the sequence length τ)?

5.To condition an RNN on a single fixed vector x (e.g. image captioning, fig 10.9), the book adds a new weight matrix R. What does R do, and what is the vector-input RNN's conditional-independence assumption (fig 10.10)?

5 questions