Ablations: three changes, three seeds each
Why it matters "This change helped" is the most common claim in any project report, and the one most often made from a single run. Module 22's rule decides it here, on your runs.
Pick three changes to your baseline, each changing exactly one setting (width, lr, steps, batch or weight_decay). Run each with seeds 0, 1 and 2, and the baseline again with the same seeds (in this lesson's log, so every number is in one place). Use evals=1: only the final score is needed.
For each change: diff = mean score of the change minus mean score of the baseline (higher is better), and se = judge.standard_error(baseline, change). Verdict: "better" if diff > 2 se, "worse" if diff < -2 se, otherwise "not shown".
The total must fit your budget: a change that costs more (wider, more steps) uses more of it.
Words and symbols in this lesson
- baseline
- the plain run every change is measured against, over the same seeds
- ablation
- one change to the baseline, run over several seeds, to see what it does
Set CHANGES (three, each one setting). Fill log with the 12 runs and verdict[name] for each change. Print the means and 2 se.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.