Finding a learning rate fast, and clipping spikes
Why it matters Guessing learning rates wastes days. One short run tells you a good one. And one bad batch can blow up a run unless you cap the step.
The range test
Train for a few hundred steps while raising the learning rate a little every step, from tiny to huge, and record the loss. At first nothing happens; then the loss falls fast; then it explodes. A good learning rate sits where the loss was falling fastest, a bit below the explosion. A common rule: take the rate where the loss was lowest and divide by 10.
Clipping
If the total size of the gradient (its length, or norm) is above a limit, scale the whole gradient down to that length. The direction is kept; only a rare huge step is tamed.
Gradient norm: the length of all the gradients taken as one long vector: the square root of the sum of their squares.
Words and symbols in this lesson
- gradient norm, clipping
- the total size of the gradient, and capping it
Write pick_lr(lrs, losses): the learning rate where the loss was lowest, divided by 10. And clip(grads, max_norm): grads is a list of arrays; if their combined norm is above max_norm, return them all scaled so the norm equals max_norm, else return them unchanged.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.