Error bars by resampling (bootstrap)
Why it matters "95% accuracy on 100 items" could easily be 90% or 99% on the next 100. An error bar says how far to trust the number.
See it
Play the animation. You only have one test set, but you can ask: if the test set had come out a little differently, how much would the score change? Make a new pretend test set by drawing 100 items from your 100 at random, with replacement (some items twice, some not at all), and score it. Do that 2,000 times. The middle 95% of those scores is your error bar.
Bootstrap: estimating uncertainty by resampling your own data with replacement. 95% interval: the 2.5th to 97.5th percentile of the resampled scores.
Words and symbols in this lesson
- bootstrap
- resampling your test set with repeats to see how much a score wobbles
- 95% interval
- the middle 95% of the resampled scores
Write bootstrap_ci(correct, n=2000, seed=0): draw all resamples at once with np.random.default_rng(seed).integers(0, len(correct), (n, len(correct))), score each, and return the 2.5th and 97.5th percentiles.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.