Syllabus Lesson 47 of 246 · Module 3: Neuron to MLP
Module 3: Neuron to MLP

Softmax and cross-entropy together

Why it matters Every classifier in this course ends with softmax plus cross-entropy. Their combined gradient is one of the most useful results in deep learning, and it is short.

Tiny numbers

Logits [2, 1, 0], softmax [0.665, 0.245, 0.090], true class 0 (one-hot [1, 0, 0]). The gradient of the loss with respect to the logits is simply softmax minus one-hot: [-0.335, 0.245, 0.090]. In words: push the true class's score up by how much it fell short, and push every other score down by how much probability it wrongly took.

One-hot: a vector of zeros with a single 1 at the true class.

For a batch, average over rows: (softmax(Z) - onehot) / n.

Recap Gradient = prediction minus truth, again.

Words and symbols in this lesson
one-hot
zeros with a single 1 at the true class
gradient check
comparing your gradient formula with a nudge-and-measure estimate
Your turn

Write loss_and_grad(Z, labels) returning the mean cross-entropy and its gradient with respect to Z (shape (batch, classes)). Use a stable softmax. Check it yourself with your gradient checker before submitting.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.