Syllabus Lesson 104 of 246 · Module 9: Is it really better?
Module 9: Is it really better?

Beyond accuracy

Why it matters On a rare-disease test where 95% of people are healthy, a model that always says "healthy" scores 95% accuracy and is useless.

A confusion matrix counts, for every true class, how often each class was predicted: row = truth, column = prediction. From it, for one class you care about:

  • Precision: of the times the model said "positive", how often was it right?
  • Recall: of the real positives, how many did the model find?

Tiny numbers

100 people, 5 sick. The lazy model says healthy for all: accuracy 95%, recall 0 of 5 = 0. A real model flags 8 people, 4 of them sick: precision 4/8 = 0.5, recall 4/5 = 0.8, and accuracy (4 + 91) / 100 = 95%. Same accuracy, completely different usefulness.

Words and symbols in this lesson
confusion matrix
counts of true class (row) against predicted class (column)
precision, recall
of what it flagged, how much was right; of what was there, how much it found
Your turn

Write confusion(y_true, y_pred, k) (a k by k array of counts) and precision_recall(y_true, y_pred, positive) returning a pair (use 0.0 when there is nothing to divide by).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.