Does 80% sure mean right 80% of the time?
Why it matters A model's confidence is only useful if it is honest. A doctor's assistant that says "90% sure" and is right half the time is dangerous.
A model is calibrated if, among all the predictions it makes with about 80% confidence, about 80% are right. To check: sort predictions into bins by confidence (0 to 0.1, 0.1 to 0.2, and so on); in each bin compare the average confidence with the actual accuracy. Plotting one against the other is a reliability diagram; perfect calibration is the diagonal.
Expected calibration error (ECE) sums the gap in each bin, weighted by how many predictions fell in it: sum over bins of (count / total) * |accuracy - confidence|.
Tiny numbers
Four predictions at confidence 0.9, of which 2 are right: that bin's gap is |0.5 - 0.9| = 0.4. If they are all the predictions, ECE = 0.4.
Words and symbols in this lesson
- calibration, ECE
- whether confidence matches accuracy; the weighted average gap
Write ece(confidence, correct, bins=10) with equal-width bins from 0 to 1 (a confidence of exactly 1.0 goes in the top bin; skip empty bins).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.