Vanishing and exploding gradients, measured
Why it matters You just saw the gradient for an early step multiplied by w once per step. With many steps, that product either shrinks to nothing or blows up. This is the main weakness of plain RNNs.
With a weight matrix W instead of a number, going back one step multiplies the gradient by W transposed (ignoring the tanh, which can only shrink it further). One step can grow a vector by at most W's largest stretch, its spectral norm. Over many steps what counts is the spectral radius, the size of W's largest eigenvalue (Module 2, "How spread adds up, and what a matrix stretches"): after k steps the size has changed by roughly r to the power k. (For the rotations used below, the two are the same number, r.) r = 0.9 over 50 steps: 0.005. r = 1.1 over 50 steps: 117.
Two different cures
Exploding gradients have a crude cure: clip them (Module 5). Vanishing gradients have no such cure. The total gradient for a weight is a sum over every distance back in time; when the long-range terms shrink like 0.9 to the power 50, they are drowned by the short-range ones, and scaling the whole sum up scales the noise with them. The network simply cannot tell that something far back mattered. The cure there is a different architecture: the LSTM, next.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- vanishing / exploding gradient
- a gradient multiplied by the same matrix many times shrinks to nothing or blows up
- LSTM
- an RNN with a memory cell c and three gates: forget, input, output
Write gradient_sizes(W, steps): start from a gradient of ones, multiply by W.T once per step, and return the list of norms (one per step). Then set verdicts to "vanish", "steady" or "explode" for the three matrices, judging by the norm after 50 steps (below 0.01, between 0.01 and 100, above 100).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.