Softmax and cross-entropy together
Why it matters Every classifier in this course ends with softmax plus cross-entropy. Their combined gradient is one of the most useful results in deep learning, and it is short.
Tiny numbers
Logits [2, 1, 0], softmax [0.665, 0.245, 0.090], true class 0 (one-hot [1, 0, 0]). The gradient of the loss with respect to the logits is simply softmax minus one-hot: [-0.335, 0.245, 0.090]. In words: push the true class's score up by how much it fell short, and push every other score down by how much probability it wrongly took.
One-hot: a vector of zeros with a single 1 at the true class.
For a batch, average over rows: (softmax(Z) - onehot) / n.
Recap Gradient = prediction minus truth, again.
Words and symbols in this lesson
- one-hot
- zeros with a single 1 at the true class
- gradient check
- comparing your gradient formula with a nudge-and-measure estimate
Write loss_and_grad(Z, labels) returning the mean cross-entropy and its gradient with respect to Z (shape (batch, classes)). Use a stable softmax. Check it yourself with your gradient checker before submitting.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.