Syllabus Lesson 36 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

The gradient of softmax, and why the loss hides it

Why it matters Softmax sits inside attention, where its gradient is needed on its own. Combined with cross-entropy it collapses to something very simple, and knowing why is what lets you trust the simple version.

The full gradient

For p = softmax(z), each output depends on every score. Nudging zj moves pi by pi (1 - pi) times the nudge when i = j, and by - pi pj when i is not j: raising one score pulls probability away from all the others. As one table (the Jacobian): diag(p) - p pT.

Backward never builds that table. Given g = dL/dp, the gradient with respect to z is p * (g - (g · p)), item by item: each pi times how much its own g beats the probability-weighted average of g.

Tiny numbers

p = [0.5, 0.3, 0.2] and g = [1, 0, 0]. Then g · p = 0.5, and dL/dz = [0.5 × 0.5, 0.3 × -0.5, 0.2 × -0.5] = [0.25, -0.15, -0.10]. The entries add to 0: adding the same amount to every score changes nothing.

Why cross-entropy makes it simple

With L = -log py, g is -1 / py at the true class and 0 elsewhere. Put that into p * (g - (g · p)) and it becomes p - onehot(y): the formula Module 3 used. The simple form is this same gradient, not a shortcut.

Recap Softmax backward: p * (g - g · p). With cross-entropy on top: p - onehot.

Words and symbols in this lesson
softmax
exp of each score divided by the total: scores become probabilities
cross-entropy
minus the log of the probability given to the right answer
Your turn

Write softmax_jacobian(z) (a 1D array of scores in, the (n, n) table out) and softmax_backward(z, g) for 2D arrays (one row per example), using p * (g - (g · p)) row by row, without building the table. The hidden test checks both against nudging and checks that with cross-entropy's g the answer is p - onehot.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.