Syllabus Lesson 70 of 246 · Module 5: Optimisation
Module 5: Optimisation

Adam: the ball plus the knobs

Why it matters Adam is the default optimiser for most deep learning. It is momentum and RMSProp together, plus one small correction that matters a lot at the start.

Tiny numbers first

One weight, beta1 = 0.9, beta2 = 0.999, lr = 0.1. Gradient at step 1 is 2.0. The running average of gradients, m, starts at 0: m = 0.9 * 0 + 0.1 * 2 = 0.2. But the true recent gradient is 2, not 0.2: m is biased towards its starting zero. Divide by (1 - 0.9 to the power t) to fix it: 0.2 / 0.1 = 2.0. The same correction for the squared average v: 0.004 / 0.001 = 4.0. Step = 0.1 * 2 / sqrt(4) = 0.1.

Without the correction the first step would be 0.1 * 0.2 / sqrt(0.004) = 0.32, more than three times too big, and in a real network those wild first steps can wreck the start of training.

m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g ** 2
m_hat = m / (1 - b1 ** t)
v_hat = v / (1 - b2 ** t)
p = p - lr * m_hat / (np.sqrt(v_hat) + eps)

Bias correction: dividing by (1 - beta to the t) so averages that started at zero are not underestimated in early steps. t counts steps from 1.

Words and symbols in this lesson
momentum (beta)
keeping part of the last step's velocity
RMSProp
dividing each weight's step by how loud its gradient usually is
Adam (m, v, t)
momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
Your turn

First fill in first_step_with and first_step_without: the size of step 1 in the example above, with and without bias correction. Then write the class Adam(lr, b1=0.9, b2=0.999, eps=1e-8) with a method step(p, g) that returns the updated parameter array.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.