The gradient of softmax, and why the loss hides it
Why it matters Softmax sits inside attention, where its gradient is needed on its own. Combined with cross-entropy it collapses to something very simple, and knowing why is what lets you trust the simple version.
The full gradient
For p = softmax(z), each output depends on every score. Nudging zj moves pi by pi (1 - pi) times the nudge when i = j, and by - pi pj when i is not j: raising one score pulls probability away from all the others. As one table (the Jacobian): diag(p) - p pT.
Backward never builds that table. Given g = dL/dp, the gradient with respect to z is p * (g - (g · p)), item by item: each pi times how much its own g beats the probability-weighted average of g.
Tiny numbers
p = [0.5, 0.3, 0.2] and g = [1, 0, 0]. Then g · p = 0.5, and dL/dz = [0.5 × 0.5, 0.3 × -0.5, 0.2 × -0.5] = [0.25, -0.15, -0.10]. The entries add to 0: adding the same amount to every score changes nothing.
Why cross-entropy makes it simple
With L = -log py, g is -1 / py at the true class and 0 elsewhere. Put that into p * (g - (g · p)) and it becomes p - onehot(y): the formula Module 3 used. The simple form is this same gradient, not a shortcut.
Recap Softmax backward: p * (g - g · p). With cross-entropy on top: p - onehot.
Words and symbols in this lesson
- softmax
- exp of each score divided by the total: scores become probabilities
- cross-entropy
- minus the log of the probability given to the right answer
Write softmax_jacobian(z) (a 1D array of scores in, the (n, n) table out) and softmax_backward(z, g) for 2D arrays (one row per example), using p * (g - (g · p)) row by row, without building the table. The hidden test checks both against nudging and checks that with cross-entropy's g the answer is p - onehot.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.