Softmax and cross-entropy in one step
Why it matters You could build softmax from exp, sum and divide, but the result is slower and can overflow. Every real library fuses softmax and cross-entropy into one operation with the simple gradient you derived in Module 3.
A fused operation computes its forward in one go (with the max-shift for safety) and has one backward rule: (softmax - onehot) / n, times the arriving gradient. You also need relu for a network: pass the gradient through where the input was positive.
Recap Fuse what you can; the combined gradient is simpler than its parts.
Words and symbols in this lesson
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- .grad
- the gradient of the final loss with respect to this Tensor
- unbroadcast
- summing a gradient back down to the shape it was stretched from
Write relu and cross_entropy(labels) (on a (batch, classes) Tensor of logits, returning a scalar Tensor) with their backward rules.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.