One seed is an anecdote
Why it matters Training the same model twice with different random seeds gives different results. A difference between two models smaller than that spread is not a result, and a single run cannot tell you how big the spread is.
A claim: "at 8,000 tokens, width 1024 beats width 512", backed by one run each (seed 0: 2.793 against 2.799). The course recorded 3 seeds of each.
A simple rule for "A beats B" with a few seeds each: take the difference of the means, and its standard error (how much that difference would move if you reran the seeds), sqrt(var(A) / nA + var(B) / nB) with the sample variance (ddof=1). Call it shown when the difference is more than 2 standard errors. judge.standard_error(a, b) computes it.
Predict first: with three seeds, does width 1024 still beat 512 at 8,000 tokens?
Words and symbols in this lesson
- seed
- the number that fixes a run's random choices; another seed gives another result
- standard error
- how much a measured difference would move if the seeds were rerun
Set my_guess to "yes" or "no". For 8,000 and 1,500,000 tokens, fill diff[tokens] (mean loss of width 512 minus mean loss of width 1024: positive means 1024 is better) and two_se[tokens]. Set verdict[tokens] to "shown" or "not shown".
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.