Syllabus Lesson 82 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

The train/eval trap

Why it matters A model that scores well in testing and badly in use, for no visible reason, is very often this bug. It has fooled experienced engineers.

A small classifier has a BatchNorm layer. It was trained properly and its running averages are good. On the test set, scored in big batches, it does well. Used on one example at a time, it is wrong far too often.

Detective work first: print what the BatchNorm layer outputs for two different examples, and ask yourself what statistics it is normalising with. Work it out from what you print; the three hints walk you there if you get stuck.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
BatchNorm (gamma, beta)
normalise each feature over the batch, then rescale with learned gamma and beta
Your turn

predict_one(x) is used on one example at a time, and its accuracy is poor. Find out why, then fix predict_one so held-out accuracy is at least 90%. Set cause to a short phrase naming the cause.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.