Bias and variance, and weight decay as a prior
Why it matters Overfitting shows up as a gap: training error far below the error on new data. Two measurable causes sit behind every such gap, and they need different cures.
Draw five fresh training sets from the same source and fit the same kind of model to each. Look at the five predictions at one point:
- If they all agree but are all wrong in the same way: high bias (too simple).
- If they disagree wildly with each other: high variance (too flexible, chasing each set's noise).
Variance is how much the model changes when the training data changes. High variance is the usual cause of overfitting, but the two words are not the same thing: overfitting is the gap you see, variance is one thing you can measure behind it.
Weight decay as a prior
Weight decay adds alpha * (sum of weights squared) to the loss. In probability terms it says, before seeing data, "I believe small weights are more likely". That belief pulls wild fits towards calm ones and lowers variance, at the cost of a little bias. For polynomial fitting it has an exact solution called ridge regression: w = solve(A.T A + alpha I, A.T y), where A holds the powers of x.
Prior: what you believe about the weights before seeing data. Weight decay is a prior that favours small weights.
Words and symbols in this lesson
- overfitting, underfitting
- fitting the noise in the training data; or too simple to fit the pattern
- bias, variance
- being wrong in the same way every time; or changing a lot with the training data
- prior
- what you believe about the weights before seeing data
- ridge (alpha)
- least squares with weight decay of strength alpha
Write ridge_poly(x, y, deg, alpha) returning the coefficients (lowest power first, using the formula above), and spread_at_half(deg, alpha): fit five training sets (seeds 0 to 4, 20 points each, made by make_set) and return the standard deviation of the five predictions at x = 0.5. Then set high_variance to whichever of degrees 1, 3 or 12 (alpha 0) has the largest spread.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.