Syllabus Lesson 232 of 246 · Module 22: Judging claims
Module 22: Judging claims

One seed is an anecdote

Why it matters Training the same model twice with different random seeds gives different results. A difference between two models smaller than that spread is not a result, and a single run cannot tell you how big the spread is.

A claim: "at 8,000 tokens, width 1024 beats width 512", backed by one run each (seed 0: 2.793 against 2.799). The course recorded 3 seeds of each.

A simple rule for "A beats B" with a few seeds each: take the difference of the means, and its standard error (how much that difference would move if you reran the seeds), sqrt(var(A) / nA + var(B) / nB) with the sample variance (ddof=1). Call it shown when the difference is more than 2 standard errors. judge.standard_error(a, b) computes it.

Predict first: with three seeds, does width 1024 still beat 512 at 8,000 tokens?

Words and symbols in this lesson
seed
the number that fixes a run's random choices; another seed gives another result
standard error
how much a measured difference would move if the seeds were rerun
Your turn

Set my_guess to "yes" or "no". For 8,000 and 1,500,000 tokens, fill diff[tokens] (mean loss of width 512 minus mean loss of width 1024: positive means 1024 is better) and two_se[tokens]. Set verdict[tokens] to "shown" or "not shown".

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.