Backprop through time is ordinary backprop
Why it matters "Backpropagation through time" sounds like a new algorithm. It is not: unroll the loop and it is backprop on a deep network whose layers share one set of weights.
See it
Play the animation. Three steps of an RNN, drawn side by side, are three layers. The same weight w appears in each. Backward, the gradient for w collects one contribution from every copy, exactly like a node used several times in your Module 4 engine.
Tiny numbers
A one-number linear RNN: h_t = w h_(t-1) + x_t, starting at h_0 = 0, with loss = h_3. Inputs 1, 2, 3 and w = 0.5: h1 = 1, h2 = 2.5, h3 = 4.25. dL/dh3 = 1. Going back, dL/dh2 = w = 0.5 and dL/dh1 = w squared = 0.25. Each copy of w contributes (gradient at its output) times (the h it multiplied): 1 times h2 + 0.5 times h1 + 0.25 times h0 = 2.5 + 0.5 + 0 = 3.0.
BPTT: backpropagation through time; unroll the steps and backpropagate, adding up the gradient from every step that shares a weight.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- backprop through time (BPTT)
- unroll the steps and backpropagate, adding the gradient from every step that shares a weight
For the one-number RNN h_t = w * h_(t-1) + x_t with h_0 = 0 and loss h_T (the last state), write bptt(xs, w) returning dL/dw by walking backward through the steps.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.