Cross-entropy is likelihood in disguise
Why it matters Classifiers are trained with a loss called cross-entropy. It sounds new, but it is the likelihood idea from two lessons ago.
Tiny numbers
A model says the probabilities of digits 0, 1 and 2 are [0.665, 0.245, 0.090]. The true digit is 0. How likely did the model think the truth was? 0.665. Under maximum likelihood we want that number big; as a loss we want something small, so we take minus its log: -log(0.665) = 0.408.
If the true digit were 2, the loss would be -log(0.090) = 2.41: the model gave the truth only 9%, so it is punished hard. A perfect prediction (probability 1) has loss 0.
Cross-entropy loss: minus the log of the probability the model gave to the true answer, averaged over examples. It is exactly the negative log-likelihood.
Picking one entry per row
With a batch you need, for each row, the probability in that row's true column. NumPy does this with two index arrays of the same length: probs[rows, cols] picks probs[rows[0], cols[0]], probs[rows[1], cols[1]] and so on. With probs = [[0.7, 0.3], [0.2, 0.8]] and labels [0, 1], probs[np.arange(2), labels] gives [0.7, 0.8]: row 0 column 0, then row 1 column 1. This is called fancy indexing.
A useful landmark: a model that guesses uniformly over 10 classes gives each 0.1, so its loss is -log(0.1) = 2.30. A fresh classifier starts near there.
Recap Loss = -log(probability given to the truth).
Words and symbols in this lesson
- likelihood
- how probable the data you saw is under one setting of the model
- cross-entropy
- minus the log of the probability given to the right answer
Write cross_entropy(probs, labels): probs is (batch, classes) of probabilities, labels holds the true class of each row; return the mean of -log(probability of the true class). No loops. Then set uniform_10_loss to the loss of a uniform guess over 10 classes.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.