Syllabus Lesson 72 of 246 · Module 5: Optimisation
Module 5: Optimisation

Warmup and cosine schedules

Why it matters The best learning rate is not the same all the way through training. Big models are almost always trained with a schedule: a gentle start, a long high middle, and a slow landing.

See it

Play the animation. Warmup: the rate climbs in a straight line from near zero to its peak over the first few hundred steps, because at the very start the gradients are large and unreliable. Cosine decay: then it slides down along half a cosine wave to a small floor, so late training takes small careful steps and settles into a good spot.

if step < warmup:  lr = peak * (step + 1) / warmup
else:              progress = (step - warmup) / (total - warmup)
                   lr = floor + (peak - floor) * 0.5 * (1 + cos(pi * progress))

Recap Ramp up, cruise, glide down.

Words and symbols in this lesson
warmup, cosine
start the learning rate small, then glide it down along a cosine
Your turn

Write lr_at(step, total, warmup, peak, floor) following the formulas above (steps count from 0).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.