Syllabus Lesson 241 of 246 · Module 23: Capstone studio: a model card you can run
Module 23: Capstone studio: a model card you can run

Held-out evaluation, with error bars

Why it matters Every choice you made looked at the validation score, so it is now a little optimistic. One evaluation on data you never looked at, with error bars, is the number you can report.

Choose your FINAL settings from what your ablations showed, at most 4 baseline costs. Train it with seed 0. Then score it once on the hidden test set: track.score(model, hidden=True, per_item=True) gives one number per test item, and studio.interval(items) the mean and a 95% bootstrap interval.

The hidden set ships inside this page, so it is hidden, not sealed: you could look at it. The course's certificate does not rest on its secrecy; your card does, so look once, at the end.

Pass marks (the default plan clears both): language, mean at least -2.72; images, at least 0.42.

Words and symbols in this lesson
baseline
the plain run every change is measured against, over the same seeds
ablation
one change to the baseline, run over several seeds, to see what it does
hidden test set
data scored once, at the end, after every choice was made
Your turn

Set FINAL, train final, and fill validation (its validation score), hidden (mean, low, high) and gap (validation minus hidden mean).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.