A test set the model has already seen
Why it matters Large models train on most of the public internet, and public test sets are on the internet. When test items leak into training, scores go up for a reason that has nothing to do with skill, and the paper reporting them may never know.
A public test set of 120 passages for the course GPT. Its authors say it was written separately. Some of it was in fact copied from the course's lesson text, which is what the GPT trained on.
A standard check: cut each passage into 13-character pieces and look them up in the training text. A passage written separately shares few long pieces by chance; a copied one shares all of them. judge.overlap(p) gives the share of a passage's pieces found in the training text, and judge.losses(model, passages) the model's loss on each passage.
Predict first: will removing the leaked passages make the reported loss better or worse?
Words and symbols in this lesson
- contamination (leak)
- test items that were also in the training data
Set my_guess to "better" or "worse". Fill reported (mean loss over all 120), leaked (a list of True/False, overlap above 0.5), clean (mean loss over the passages that are not leaked) and leaked_loss.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.