Syllabus Lesson 127 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

Backprop through time is ordinary backprop

Why it matters "Backpropagation through time" sounds like a new algorithm. It is not: unroll the loop and it is backprop on a deep network whose layers share one set of weights.

See it

Play the animation. Three steps of an RNN, drawn side by side, are three layers. The same weight w appears in each. Backward, the gradient for w collects one contribution from every copy, exactly like a node used several times in your Module 4 engine.

Tiny numbers

A one-number linear RNN: h_t = w h_(t-1) + x_t, starting at h_0 = 0, with loss = h_3. Inputs 1, 2, 3 and w = 0.5: h1 = 1, h2 = 2.5, h3 = 4.25. dL/dh3 = 1. Going back, dL/dh2 = w = 0.5 and dL/dh1 = w squared = 0.25. Each copy of w contributes (gradient at its output) times (the h it multiplied): 1 times h2 + 0.5 times h1 + 0.25 times h0 = 2.5 + 0.5 + 0 = 3.0.

BPTT: backpropagation through time; unroll the steps and backpropagate, adding up the gradient from every step that shares a weight.

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
backprop through time (BPTT)
unroll the steps and backpropagate, adding the gradient from every step that shares a weight
Your turn

For the one-number RNN h_t = w * h_(t-1) + x_t with h_0 = 0 and loss h_T (the last state), write bptt(xs, w) returning dL/dw by walking backward through the steps.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.