Held-out evaluation, with error bars
Why it matters Every choice you made looked at the validation score, so it is now a little optimistic. One evaluation on data you never looked at, with error bars, is the number you can report.
Choose your FINAL settings from what your ablations showed, at most 4 baseline costs. Train it with seed 0. Then score it once on the hidden test set: track.score(model, hidden=True, per_item=True) gives one number per test item, and studio.interval(items) the mean and a 95% bootstrap interval.
The hidden set ships inside this page, so it is hidden, not sealed: you could look at it. The course's certificate does not rest on its secrecy; your card does, so look once, at the end.
Pass marks (the default plan clears both): language, mean at least -2.72; images, at least 0.42.
Words and symbols in this lesson
- baseline
- the plain run every change is measured against, over the same seeds
- ablation
- one change to the baseline, run over several seeds, to see what it does
- hidden test set
- data scored once, at the end, after every choice was made
Set FINAL, train final, and fill validation (its validation score), hidden (mean, low, high) and gap (validation minus hidden mean).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.