Syllabus Lesson 73 of 246 · Module 5: Optimisation
Module 5: Optimisation

Finding a learning rate fast, and clipping spikes

Why it matters Guessing learning rates wastes days. One short run tells you a good one. And one bad batch can blow up a run unless you cap the step.

The range test

Train for a few hundred steps while raising the learning rate a little every step, from tiny to huge, and record the loss. At first nothing happens; then the loss falls fast; then it explodes. A good learning rate sits where the loss was falling fastest, a bit below the explosion. A common rule: take the rate where the loss was lowest and divide by 10.

Clipping

If the total size of the gradient (its length, or norm) is above a limit, scale the whole gradient down to that length. The direction is kept; only a rare huge step is tamed.

Gradient norm: the length of all the gradients taken as one long vector: the square root of the sum of their squares.

Words and symbols in this lesson
gradient norm, clipping
the total size of the gradient, and capping it
Your turn

Write pick_lr(lrs, losses): the learning rate where the loss was lowest, divided by 10. And clip(grads, max_norm): grads is a list of arrays; if their combined norm is above max_norm, return them all scaled so the norm equals max_norm, else return them unchanged.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.