A heavy ball (momentum)
Why it matters Momentum is the single most useful fix for stretched valleys, and almost every optimiser you will use has it.
See it first
Play the animation. Plain gradient descent zigzags across the valley. A heavy ball keeps some of its previous velocity: the zigzags cancel out (left, right, left) while the steady downhill pull along the valley adds up step after step. So it speeds along the valley floor.
Name it
v = beta * v + grad # velocity: mostly the old velocity, plus the new push
p = p - lr * v
With beta = 0.9 the ball remembers roughly the last 10 pushes.
Why plain descent is slow here
On this bowl the steep direction is 50 times steeper than the gentle one. Plain descent's learning rate is capped by the steep direction: above 2/50 = 0.04 the steps across the valley grow and it blows up. Just under the cap, at 0.039, it zigzags across the valley (each step overshoots and lands on the other side) while creeping along it, because 0.039 is a tiny rate for the gentle direction. Momentum can use a larger rate and keep its speed along the valley: it overshoots in the steep direction too, but those swings cancel while the pushes along the floor add up.
(At exactly lr = 1/50 = 0.02 there is no zigzag at all: one step lands on the valley floor, and the rest is the slow crawl. Try it.)
(The scene draws a gentler valley, 25 times steeper across, with beta 0.5, so its paths fit on the screen; the numbers in this lesson are for the 50 times valley you code.)
Momentum: a running sum of recent gradients that smooths out zigzags and builds speed in consistent directions.
Words and symbols in this lesson
- momentum (beta)
- keeping part of the last step's velocity
Write steps_plain(lr) and steps_momentum(lr, beta): starting at (1, 1) on the stretched bowl, count steps until the distance to the bottom is below 0.001 (max 20,000). Give each its own good settings: plain descent at lr 0.039, just under its cap, and momentum at lr 0.06 with beta 0.6. Momentum must need at most a third of the steps.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.