Adam: the ball plus the knobs
Why it matters Adam is the default optimiser for most deep learning. It is momentum and RMSProp together, plus one small correction that matters a lot at the start.
Tiny numbers first
One weight, beta1 = 0.9, beta2 = 0.999, lr = 0.1. Gradient at step 1 is 2.0. The running average of gradients, m, starts at 0: m = 0.9 * 0 + 0.1 * 2 = 0.2. But the true recent gradient is 2, not 0.2: m is biased towards its starting zero. Divide by (1 - 0.9 to the power t) to fix it: 0.2 / 0.1 = 2.0. The same correction for the squared average v: 0.004 / 0.001 = 4.0. Step = 0.1 * 2 / sqrt(4) = 0.1.
Without the correction the first step would be 0.1 * 0.2 / sqrt(0.004) = 0.32, more than three times too big, and in a real network those wild first steps can wreck the start of training.
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g ** 2
m_hat = m / (1 - b1 ** t)
v_hat = v / (1 - b2 ** t)
p = p - lr * m_hat / (np.sqrt(v_hat) + eps)
Bias correction: dividing by (1 - beta to the t) so averages that started at zero are not underestimated in early steps. t counts steps from 1.
Words and symbols in this lesson
- momentum (beta)
- keeping part of the last step's velocity
- RMSProp
- dividing each weight's step by how loud its gradient usually is
- Adam (m, v, t)
- momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
First fill in first_step_with and first_step_without: the size of step 1 in the example above, with and without bias correction. Then write the class Adam(lr, b1=0.9, b2=0.999, eps=1e-8) with a method step(p, g) that returns the updated parameter array.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.