Syllabus Lesson 34 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

Scores into probabilities (softmax)

Why it matters A digit classifier ends with 10 raw scores, one per digit. To read them as "how likely is each digit", they must be positive and add up to 1.

Tiny numbers

Scores [2, 1, 0]. Make them positive with exp: about [7.39, 2.72, 1.00]. Divide by the total, 11.11: [0.665, 0.245, 0.090]. They add up to 1, keep the same order, and a higher score always means a higher probability.

Softmax: exp of each score, divided by the sum of all the exps. Logits: the raw scores that go in.

One trap: adding the same number to every score changes nothing (it cancels in the division). Try [1002, 1001, 1000]: same answer as [2, 1, 0] on paper, but exp(1002) overflows on a computer. You will fix that in the gate.

Recap exp, then divide by the total.

Words and symbols in this lesson
softmax
exp of each score divided by the total: scores become probabilities
Your turn

Write softmax(z) for a 1D array (the plain version is fine here), and shift_invariant: set it to True if adding the same number to every score leaves the output the same, or False if it changes it.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.