Syllabus Lesson 35 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

Cross-entropy is likelihood in disguise

Why it matters Classifiers are trained with a loss called cross-entropy. It sounds new, but it is the likelihood idea from two lessons ago.

Tiny numbers

A model says the probabilities of digits 0, 1 and 2 are [0.665, 0.245, 0.090]. The true digit is 0. How likely did the model think the truth was? 0.665. Under maximum likelihood we want that number big; as a loss we want something small, so we take minus its log: -log(0.665) = 0.408.

If the true digit were 2, the loss would be -log(0.090) = 2.41: the model gave the truth only 9%, so it is punished hard. A perfect prediction (probability 1) has loss 0.

Cross-entropy loss: minus the log of the probability the model gave to the true answer, averaged over examples. It is exactly the negative log-likelihood.

Picking one entry per row

With a batch you need, for each row, the probability in that row's true column. NumPy does this with two index arrays of the same length: probs[rows, cols] picks probs[rows[0], cols[0]], probs[rows[1], cols[1]] and so on. With probs = [[0.7, 0.3], [0.2, 0.8]] and labels [0, 1], probs[np.arange(2), labels] gives [0.7, 0.8]: row 0 column 0, then row 1 column 1. This is called fancy indexing.

A useful landmark: a model that guesses uniformly over 10 classes gives each 0.1, so its loss is -log(0.1) = 2.30. A fresh classifier starts near there.

Recap Loss = -log(probability given to the truth).

Words and symbols in this lesson
likelihood
how probable the data you saw is under one setting of the model
cross-entropy
minus the log of the probability given to the right answer
Your turn

Write cross_entropy(probs, labels): probs is (batch, classes) of probabilities, labels holds the true class of each row; return the mean of -log(probability of the true class). No loops. Then set uniform_10_loss to the loss of a uniform guess over 10 classes.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.