Syllabus Lesson 71 of 246 · Module 5: Optimisation
Module 5: Optimisation

AdamW: weight decay done right

Why it matters Most large models are trained with AdamW, not Adam. The difference is one line, and it changes what "weight decay" actually does.

Weight decay pulls every weight a little towards zero each step, which discourages large weights and usually helps the model work on new data.

  • Adam with L2 adds wd * p to the gradient before Adam's scaling. Adam then divides it by the volume knob, so weights with loud gradients barely decay at all.
  • AdamW leaves the gradient alone and decays the weight directly: p_new = p - lr * (adam_direction + wd * p), where p is the weight before this step. Every weight decays by the same proportion.

Decoupled weight decay: shrinking the weights directly instead of through the gradient. That is the "W" in AdamW.

Words and symbols in this lesson
Adam (m, v, t)
momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
AdamW, weight decay
shrinking weights directly a little each step
Your turn

Write adam_l2_step(p, g, state, lr, wd) and adamw_step(p, g, state, lr, wd). state is a dictionary holding m, v and t (start with an empty dictionary). Use b1 = 0.9, b2 = 0.999, eps = 1e-8. Each returns the new p. For AdamW the update is p_new = p - lr * (m_hat / (sqrt(v_hat) + eps) + wd * p), with p the weight from before the step.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.