One rule behind every "is it real" check
Why it matters This course has judged differences in at least four ways: a paired interval (Module 9), "0.1 higher and higher on every seed" (Modules 16 and 17), and 2 standard errors (this module). They are all the same idea, and knowing it lets you pick the right one, and know when one is too generous.
Every rule compares a difference with its noise. Two choices change the answer:
- Paired or not. When both models are scored on the same items (Module 9's test images), subtract item by item first: how hard each item is cancels out, the noise shrinks, and small real differences show. For two classifiers on the same items, the cleanest count is just the items where they disagree (McNemar's test). Seeds of separate runs are not paired: compare the two groups as wholes.
- How strict the cutoff is. 2 standard errors assumes you know the noise. With 3 seeds per side you estimated it from 3 numbers each, and that estimate can easily be too small. Student's t accounts for it: the cutoff becomes
stats.t.ppf(0.975, df)standard errors, where df (the degrees of freedom) comes from Welch's formula,df = (va + vb)**2 / (va**2 / (na - 1) + vb**2 / (nb - 1))withva = var(a) / na. With 3 seeds per side df is 2 to 4, and the cutoff 2.8 to 4.3, not 2.
Predict first: on the recorded grid, does the stricter cutoff ever overturn a "shown" from the 2 standard error rule?
Words and symbols in this lesson
- seed
- the number that fixes a run's random choices; another seed gives another result
- standard error
- how much a measured difference would move if the seeds were rerun
Set my_guess to "yes" or "no". Write welch(a, b) returning (diff, se, df) for two lists of seed losses (diff = mean(a) - mean(b): positive when b, the larger model, is better). Then, for each case in CASES, fill result[case] with a dict: 'ratio' (diff / se), 'cutoff' (the t cutoff), 'two_se' and 't' ("shown" or "not shown" by each rule). Set disagree to the list of cases where the two rules differ.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.