Syllabus Lesson 103 of 246 · Module 9: Is it really better?
Module 9: Is it really better?

Splits and leakage

Why it matters If any information from the test data reaches training, the test score is no longer a test. This is the most common way honest people publish wrong numbers.

The rule: the test set must play no part in any decision, including preprocessing. Three ways it sneaks in:

  • Scaling with all the data. Computing the mean and spread on every row, test rows included, lets test values shape the training inputs. Small, but real.
  • Choosing features with all the labels. Picking the features most correlated with the label, using the test labels too, can make pure noise look predictive. This one can be huge.
  • Duplicates across the split. The same example in both sets: the model is tested on something it memorised.

Leakage: test information reaching the training process, making scores look better than they really are.

Your turn

Set leaky to a dictionary saying which of the three pipelines leak. Then write pipeline_3_fixed(X, y): the same idea, choosing the 5 features using only the first 200 rows.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.