Syllabus Lesson 238 of 246 · Module 23: Capstone studio: a model card you can run
Module 23: Capstone studio: a model card you can run

Ablations: three changes, three seeds each

Why it matters "This change helped" is the most common claim in any project report, and the one most often made from a single run. Module 22's rule decides it here, on your runs.

Pick three changes to your baseline, each changing exactly one setting (width, lr, steps, batch or weight_decay). Run each with seeds 0, 1 and 2, and the baseline again with the same seeds (in this lesson's log, so every number is in one place). Use evals=1: only the final score is needed.

For each change: diff = mean score of the change minus mean score of the baseline (higher is better), and se = judge.standard_error(baseline, change). Verdict: "better" if diff > 2 se, "worse" if diff < -2 se, otherwise "not shown".

The total must fit your budget: a change that costs more (wider, more steps) uses more of it.

Words and symbols in this lesson
baseline
the plain run every change is measured against, over the same seeds
ablation
one change to the baseline, run over several seeds, to see what it does
Your turn

Set CHANGES (three, each one setting). Fill log with the 12 runs and verdict[name] for each change. Print the means and 2 se.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.