Syllabus Lesson 229 of 246 · Module 22: Judging claims
Module 22: Judging claims

A test set the model has already seen

Why it matters Large models train on most of the public internet, and public test sets are on the internet. When test items leak into training, scores go up for a reason that has nothing to do with skill, and the paper reporting them may never know.

A public test set of 120 passages for the course GPT. Its authors say it was written separately. Some of it was in fact copied from the course's lesson text, which is what the GPT trained on.

A standard check: cut each passage into 13-character pieces and look them up in the training text. A passage written separately shares few long pieces by chance; a copied one shares all of them. judge.overlap(p) gives the share of a passage's pieces found in the training text, and judge.losses(model, passages) the model's loss on each passage.

Predict first: will removing the leaked passages make the reported loss better or worse?

Words and symbols in this lesson
contamination (leak)
test items that were also in the training data
Your turn

Set my_guess to "better" or "worse". Fill reported (mean loss over all 120), leaked (a list of True/False, overlap above 0.5), clean (mean loss over the passages that are not leaked) and leaked_loss.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.