Gate: review a claim (AI off)
Why it matters Reading results critically is a skill you will use on every paper, model card and benchmark table. The rules are simple; applying them every time, without being told which one matters, is the skill.
Gate rules: AI tutor off. Write review(claim). A claim says "model A beats model B" and comes with the loss of each seed of each model and the share of the test set found in the training data. In order:
- If more than 10% of the test set was in the training data, return "leaked": no score on it can be trusted.
- If either model has fewer than 2 seeds, return "not shown": the spread is unknown.
- Let diff = mean(B) - mean(A) (positive when A is better) and se = sqrt(var(A) / nA + var(B) / nB), sample variances (
ddof=1). Return "shown" if diff > 2 se, "contradicted" if diff < -2 se, otherwise "not shown".
The check runs your function on six new claims, different on every check, including the edge cases: one seed, exactly 10% overlap, seeds that agree exactly.
Words and symbols in this lesson
- contamination (leak)
- test items that were also in the training data
- seed
- the number that fixes a run's random choices; another seed gives another result
Fill review. Try it on {'a': [1.689, 1.675, 1.689], 'b': [1.715, 1.721, 1.707], 'test_overlap': 0.0} (the recorded widths 1024 and 512 at 1.5 million tokens).
AI off: no tutor, hints or worked solution on this one. Your own thinking only.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.