Warmup and cosine schedules
Why it matters The best learning rate is not the same all the way through training. Big models are almost always trained with a schedule: a gentle start, a long high middle, and a slow landing.
See it
Play the animation. Warmup: the rate climbs in a straight line from near zero to its peak over the first few hundred steps, because at the very start the gradients are large and unreliable. Cosine decay: then it slides down along half a cosine wave to a small floor, so late training takes small careful steps and settles into a good spot.
if step < warmup: lr = peak * (step + 1) / warmup
else: progress = (step - warmup) / (total - warmup)
lr = floor + (peak - floor) * 0.5 * (1 + cos(pi * progress))
Recap Ramp up, cruise, glide down.
Words and symbols in this lesson
- warmup, cosine
- start the learning rate small, then glide it down along a cosine
Write lr_at(step, total, warmup, peak, floor) following the formulas above (steps count from 0).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.