Syllabus Lesson 106 of 246 · Module 9: Is it really better?
Module 9: Is it really better?

Error bars by resampling (bootstrap)

Why it matters "95% accuracy on 100 items" could easily be 90% or 99% on the next 100. An error bar says how far to trust the number.

See it

Play the animation. You only have one test set, but you can ask: if the test set had come out a little differently, how much would the score change? Make a new pretend test set by drawing 100 items from your 100 at random, with replacement (some items twice, some not at all), and score it. Do that 2,000 times. The middle 95% of those scores is your error bar.

Bootstrap: estimating uncertainty by resampling your own data with replacement. 95% interval: the 2.5th to 97.5th percentile of the resampled scores.

Words and symbols in this lesson
bootstrap
resampling your test set with repeats to see how much a score wobbles
95% interval
the middle 95% of the resampled scores
Your turn

Write bootstrap_ci(correct, n=2000, seed=0): draw all resamples at once with np.random.default_rng(seed).integers(0, len(correct), (n, len(correct))), score each, and return the 2.5th and 97.5th percentiles.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.