Comparing two models fairly
Why it matters Back to lesson 1. This is the right way to decide.
Both models answered the same 100 items. So do not put error bars on each score separately; put one on the difference, resampling items as pairs. Each resample picks 100 items with replacement and computes B's accuracy minus A's on those same items. If the 95% interval of that difference includes 0, the data cannot tell the models apart.
Paired comparison: comparing two models item by item on the same test set, which cancels out how hard each item is.
How lesson 1's 100 images were picked, honestly: the readers are simulated, and the course chose a draw of 100 images on which A happens to lead B by 4 points. Over 2,000 random draws of 100 images, A led by 4 points or more only 8% of the time; B came out ahead in half of them, and 14% were ties. It is a real kind of luck, chosen to make the point: a single test set of 100 can point the wrong way.
Words and symbols in this lesson
- bootstrap
- resampling your test set with repeats to see how much a score wobbles
- 95% interval
- the middle 95% of the resampled scores
- paired comparison
- comparing two models on the same items, item by item
Write paired_diff_ci(a, b, n=2000, seed=0) (interval of B minus A, resampling item indices as in the bootstrap lesson) and verdict(a, b) returning "A", "B" or "no clear difference". Run it on lesson 1's models.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.