Syllabus Lesson 69 of 246 · Module 5: Optimisation
Module 5: Optimisation

A volume knob per weight (RMSProp)

Why it matters Momentum smooths the direction. RMSProp fixes the other problem: some weights get huge gradients and others tiny ones, so one learning rate cannot suit them all.

RMSProp keeps, for each weight, a running average of its squared gradient: how loud that weight's gradient usually is. Then it divides each step by the square root of that, like turning down the volume on loud weights and up on quiet ones.

s = rho * s + (1 - rho) * grad ** 2
p = p - lr * grad / (np.sqrt(s) + eps)

On the stretched bowl, the steep direction has loud gradients and gets turned down, the gentle direction is turned up, and the bowl behaves almost as if it were round. That works here because the bowl's steep and gentle directions line up with the two weights. Tilt the valley 45 degrees, so that each weight sees a mix of both directions, and a volume knob per weight cannot separate them: RMSProp and Adam then help far less. They rescale by how loud each weight's gradient is, not by the true shape of the valley.

eps: a tiny number such as 1e-8 that stops division by zero.

Words and symbols in this lesson
momentum (beta)
keeping part of the last step's velocity
RMSProp
dividing each weight's step by how loud its gradient usually is
Adam (m, v, t)
momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
Your turn

Write rmsprop(grad, start, lr, rho, steps, eps=1e-8) returning the final position. The hidden test compares your position with the reference after a few steps.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.