AdamW: weight decay done right
Why it matters Most large models are trained with AdamW, not Adam. The difference is one line, and it changes what "weight decay" actually does.
Weight decay pulls every weight a little towards zero each step, which discourages large weights and usually helps the model work on new data.
- Adam with L2 adds
wd * pto the gradient before Adam's scaling. Adam then divides it by the volume knob, so weights with loud gradients barely decay at all. - AdamW leaves the gradient alone and decays the weight directly:
p_new = p - lr * (adam_direction + wd * p), wherepis the weight before this step. Every weight decays by the same proportion.
Decoupled weight decay: shrinking the weights directly instead of through the gradient. That is the "W" in AdamW.
Words and symbols in this lesson
- Adam (m, v, t)
- momentum (m) plus a per-weight volume knob (v), corrected for early steps (t)
- AdamW, weight decay
- shrinking weights directly a little each step
Write adam_l2_step(p, g, state, lr, wd) and adamw_step(p, g, state, lr, wd). state is a dictionary holding m, v and t (start with an empty dictionary). Use b1 = 0.9, b2 = 0.999, eps = 1e-8. Each returns the new p. For AdamW the update is p_new = p - lr * (m_hat / (sqrt(v_hat) + eps) + wd * p), with p the weight from before the step.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.