Syllabus Lesson 235 of 246 · Module 22: Judging claims
Module 22: Judging claims

Gate: review a claim (AI off)

Why it matters Reading results critically is a skill you will use on every paper, model card and benchmark table. The rules are simple; applying them every time, without being told which one matters, is the skill.

Gate rules: AI tutor off. Write review(claim). A claim says "model A beats model B" and comes with the loss of each seed of each model and the share of the test set found in the training data. In order:

  1. If more than 10% of the test set was in the training data, return "leaked": no score on it can be trusted.
  2. If either model has fewer than 2 seeds, return "not shown": the spread is unknown.
  3. Let diff = mean(B) - mean(A) (positive when A is better) and se = sqrt(var(A) / nA + var(B) / nB), sample variances (ddof=1). Return "shown" if diff > 2 se, "contradicted" if diff < -2 se, otherwise "not shown".

The check runs your function on six new claims, different on every check, including the edge cases: one seed, exactly 10% overlap, seeds that agree exactly.

Words and symbols in this lesson
contamination (leak)
test items that were also in the training data
seed
the number that fixes a run's random choices; another seed gives another result
Your turn

Fill review. Try it on {'a': [1.689, 1.675, 1.689], 'b': [1.715, 1.721, 1.707], 'test_overlap': 0.0} (the recorded widths 1024 and 512 at 1.5 million tokens).

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.