The train/eval trap
Why it matters A model that scores well in testing and badly in use, for no visible reason, is very often this bug. It has fooled experienced engineers.
A small classifier has a BatchNorm layer. It was trained properly and its running averages are good. On the test set, scored in big batches, it does well. Used on one example at a time, it is wrong far too often.
Detective work first: print what the BatchNorm layer outputs for two different examples, and ask yourself what statistics it is normalising with. Work it out from what you print; the three hints walk you there if you get stuck.
Words and symbols in this lesson
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
- BatchNorm (gamma, beta)
- normalise each feature over the batch, then rescale with learned gamma and beta
predict_one(x) is used on one example at a time, and its accuracy is poor. Find out why, then fix predict_one so held-out accuracy is at least 90%. Set cause to a short phrase naming the cause.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.