A volume knob per weight (RMSProp)
Why it matters Momentum smooths the direction. RMSProp fixes the other problem: some weights get huge gradients and others tiny ones, so one learning rate cannot suit them all.
RMSProp keeps, for each weight, a running average of its squared gradient: how loud that weight's gradient usually is. Then it divides each step by the square root of that, like turning down the volume on loud weights and up on quiet ones.
s = rho * s + (1 - rho) * grad ** 2
p = p - lr * grad / (np.sqrt(s) + eps)
On the stretched bowl, the steep direction has loud gradients and gets turned down, the gentle direction is turned up, and the bowl behaves almost as if it were round. That works here because the bowl's steep and gentle directions line up with the two weights. Tilt the valley 45 degrees, so that each weight sees a mix of both directions, and a volume knob per weight cannot separate them: RMSProp and Adam then help far less. They rescale by how loud each weight's gradient is, not by the true shape of the valley.
eps: a tiny number such as 1e-8 that stops division by zero.
Words and symbols in this lesson
- momentum (beta)
- keeping part of the last step's velocity
- RMSProp
- dividing each weight's step by how loud its gradient usually is
- Adam (m, v, t)
- momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
Write rmsprop(grad, start, lr, rho, steps, eps=1e-8) returning the final position. The hidden test compares your position with the reference after a few steps.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.