Teacher Forcing · Computing the Gradient (BPTT)

§10.2.1 Teacher forcing and networks with output recurrence

Before the gradient, one more architecture. The RNN whose only recurrence runs from the output at one step back to the hidden units at the next (fig 10.4) is a special case, and it comes with its own training trick.

It is strictly less powerful than the hidden-to-hidden RNN of fig 10.3: it cannot even simulate a universal Turing machine. The reason is that its only channel to the future is the output $o^{(t)}$ — and $o^{(t)}$ is explicitly trained to match the target. Unless you hand the network a target that already encodes the full system state, $o$ is unlikely to carry the past information the future needs. Its compensating advantage: for a loss that compares each prediction $o^{(t)}$ to its target $y^{(t)}$ , all time steps decouple, so training parallelizes — the gradient at each step is computed in isolation, because the training set already supplies the ideal previous output.

A close cousin reads a whole sequence and emits a single output at the end — a summarizer that maps a variable-length sequence to one fixed vector:

Fig 10.5 — RNN with a single output at the end. Hidden-to-hidden recurrence (W) carries the state across the whole sequence; only the final state is read out (V) to o⁽τ⁾ and compared to y⁽τ⁾. Used to produce a fixed-size summary of a sequence.

Teacher forcing. Models with output-to-hidden recurrence are trained with teacher forcing — a procedure that falls straight out of maximum likelihood. Decompose the likelihood of a two-step sequence:

$\log p\big(y^{(1)}, y^{(2)} \mid x^{(1)}, x^{(2)}\big) = \log p\big(y^{(2)} \mid y^{(1)}, x^{(1)}, x^{(2)}\big) + \log p\big(y^{(1)} \mid x^{(1)}, x^{(2)}\big)$

Look at $t = 2$ : the model is trained to maximize the probability of $y^{(2)}$ conditioned on the true previous value $y^{(1)}$ from the training set. So maximum likelihood itself dictates that during training we feed the recurrent connections the target values, not the model’s own guesses. Switch the widget between train and test to see exactly which value flows back — and drag the curriculum slider to see the standard fix for the resulting gap:

Fig 10.6 (interactive) — teacher forcing. Train time feeds the true y⁽ᵗ⁻¹⁾ back into h⁽ᵗ⁾; test time (open loop) must feed the model's own o⁽ᵗ⁻¹⁾. The curriculum slider ramps up self-generated inputs (scheduled sampling) to close the gap.

Train time: the recurrent input to h⁽ᵗ⁾ is the ground-truth y⁽ᵗ⁻¹⁾ from the dataset — this is teacher forcing. Because the true previous output is given, the steps decouple and can train in parallel (no BPTT needed for output-only recurrence).

Curriculum mitigation — scheduled sampling

feed model's own output with prob p = 0%

p = 0: pure teacher forcing — always feed the truth. Fastest to train, but most exposed to the open-loop gap.

Teacher forcing, BPTT, or both

Teacher forcing was motivated as a way to avoid BPTT in nets that lack hidden-to-hidden connections. But it can also be used on nets that do have hidden-to-hidden recurrence, as long as they also have output-to-hidden connections. The moment the hidden units depend on earlier time steps, though, BPTT becomes necessary — so some models train with both teacher forcing and BPTT.

The disadvantage of strict teacher forcing shows up when the network is later run in open-loop mode, feeding its own outputs (or samples) back as input. The inputs it then sees can differ sharply from the always-correct inputs it saw in training. Two mitigations: (1) train with free-running inputs too — unrolling the output-to-input path a few steps and predicting the correct target from the model’s own generated values, so the net learns to steer a wandering state back on course; (2) scheduled sampling (Bengio 2015) — randomly choose generated vs true values as input, using a curriculum that feeds more generated values as training proceeds.

§10.2.2 Computing the gradient — back-propagation through time

Here is the reassuring part: computing the gradient of an RNN needs no special algorithm. Just apply the generalized back-propagation of §6.5.6 to the unrolled computational graph. The nodes are the parameters $U, V, W, b, c$ and, for each step $t$ , the values $x^{(t)}, h^{(t)}, o^{(t)}, L^{(t)}$ . We compute $\nabla_N L$ for each node $N$ from the gradients of the nodes that follow it — working backward from the end of the sequence. Step through the recursion:

BPTT — the node-gradient recursion (eqs 10.17–10.21). Work backward from the loss.

\frac{\partial L}{\partial L^{(t)}} = 1

step 1/4: Seed. The total loss is the sum L = Σₜ L⁽ᵗ⁾, so each per-step loss enters the backward pass with gradient 1.

This recursion is where vanishing/exploding gradients live

Notice that going one step further back multiplies by $W^{\top}$ and a tanh Jacobian. Over many steps the gradient on an early state is a product of many such matrices — repeated multiplication that tends to vanish or explode. That is the central obstacle to learning long-range dependencies, revisited from §8.2.5 and taken up in depth in §10.7.

Parameter gradients. Once the internal-node gradients are known, the parameter gradients follow. Because $U, V, W$ are shared across every time step, care is needed: the calculus operator $\nabla_W$ accounts for all edges $W$ touches. The book introduces dummy variables $W^{(t)}$ — copies of $W$ used only at step $t$ — so a single edge’s contribution can be written, then summed over time:

Gradients on the shared weights (eqs 10.24–10.28)

\nabla_V L = \sum_t (\nabla_{o^{(t)}}L)\,h^{(t)\top}, \;\; \nabla_W L = \sum_t \operatorname{diag}\!\big(1-(h^{(t)})^2\big)(\nabla_{h^{(t)}}L)\,h^{(t-1)\top}, \;\; \nabla_U L = \sum_t \operatorname{diag}\!\big(1-(h^{(t)})^2\big)(\nabla_{h^{(t)}}L)\,x^{(t)\top}

$\sum_t$	the KEY consequence of parameter sharing — each weight matrix's gradient is the SUM of its contributions across all τ steps	sum over time
$\nabla_V L$	hidden→output: at each step, the output gradient outer-producted with the state h⁽ᵗ⁾ that produced it	\|o\|×\|h\|
$\nabla_W L$	hidden→hidden: the tanh-Jacobian-scaled hidden gradient outer-producted with the PREVIOUS state h⁽ᵗ⁻¹⁾	\|h\|×\|h\|
$\nabla_U L$	input→hidden: the same scaled hidden gradient outer-producted with the input x⁽ᵗ⁾	\|h\|×\|x\|
$\operatorname{diag}(1-(h^{(t)})^2)$	the tanh Jacobian — appears wherever the gradient passes back through the hidden nonlinearity	\|h\|×\|h\| diagonal

The biases follow the same pattern: $\nabla_c L = \sum_t \nabla_{o^{(t)}} L$ and $\nabla_b L = \sum_t \operatorname{diag}(1-(h^{(t)})^2)\,\nabla_{h^{(t)}} L$ . And note what is absent: we never compute $\nabla_{x^{(t)}} L$ for training, because the inputs have no parameters as ancestors in the loss graph.

Next: the RNN seen as a probability model — recurrent networks as directed graphical models, and conditioning on context.

Check yourself — teacher forcing and back-propagation through time

1.Why is an RNN with only output-to-hidden recurrence (fig 10.4) strictly less powerful than one with hidden-to-hidden recurrence (fig 10.3)?

2.What is teacher forcing, and where does it come from?

3.In the TeacherForcingLab, you switch from Train to Test and the recurrent input changes from the gold true y to the green model output o. What problem does this illustrate, and how is it mitigated?

4.In BPTT for the softmax-output RNN, what is the gradient of the loss on the output logits, ∇o⁽ᵗ⁾L?

5.When computing ∇_W L for the shared recurrent matrix W, why does the book introduce “dummy variables” W⁽ᵗ⁾ and sum over t?

5 questions