Syllabus Lesson 110 of 246 · Module 9: Is it really better?
Module 9: Is it really better?

Gate: evaluate a new pair of models (AI off)

Why it matters Making a sound call, with an interval, on fresh results is what this module is for.

AI tutor off. Two new models, 1,000 shared test items.

Words and symbols in this lesson
bootstrap
resampling your test set with repeats to see how much a score wobbles
Your turn

Write paired_interval(a, b), the 95% paired bootstrap interval of B minus A (2,000 resamples, seed 0, as in the last lesson), and make_call(a, b), returning "A", "B" or "no clear difference". Then set interval and call for this pair. The check also runs your two functions on three more pairs of models it makes up itself.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.