Syllabus Lesson 91 of 246 · Module 7: Working on new data
Module 7: Working on new data

Double descent

Why it matters The textbook rule "more parameters means more overfitting" breaks for big models. Seeing why changes how you think about model size.

See it

Play the animation. We fit 40 training points using only the first p of 400 available features, choosing among exact fits the one with the smallest weights. As p grows towards the number of training points (40), test error climbs, slowly and then steeply, to a peak exactly where the two are equal. There the model has just enough freedom to hit every training point, and only one way to do it, usually a wild one. Past the peak, there are many exact fits and choosing the smallest-weights one is a gentle fit: test error falls again, and with enough features it ends lower than the small models.

Interpolation threshold: the model size at which it can just fit the training data exactly. Double descent peaks there.

Call the Curve: predict the number of features with the highest test error, and whether p = 400 beats p = 5.

Read the numbers honestly

This toy is built to show the shape of the curve, not to predict well. Its signal is spread thinly over all 400 features and there are only 40 training points, so every model here is poor: predicting 0 for everything scores a median test error of about 1.01, p = 5 scores about 1.11 and p = 400 about 0.91. The peak at p = 40 is about 36. The second descent is real, but it lands only a little below "predict nothing".

The peak also has a cure you already know. Weight decay (ridge) with a strength chosen on validation data, instead of the smallest-weights exact fit, keeps the error near 1.0 at every size, p = 40 included (about 1.02). Double descent is what happens to exact fits; a tuned regulariser removes the spike.

Words and symbols in this lesson
overfitting, underfitting
fitting the noise in the training data; or too simple to fit the pattern
ridge (alpha)
least squares with weight decay of strength alpha
interpolation threshold
the model size that can just fit the training data exactly
Your turn

Fill in my_peak and my_400_beats_5. Then write test_error(p): for seeds 0 to 19, fit with np.linalg.pinv(X[:, :p]) @ y, measure the test mean squared error, and return the median over seeds.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.