Syllabus Lesson 80 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

BatchNorm forward and running averages

Why it matters At training time BatchNorm uses the batch's own statistics. When the model is used on one example at a time there is no batch, so it must remember typical statistics from training.

During training, keep running averages of each feature's mean and variance: running = (1 - momentum) * running + momentum * batch_value, with momentum 0.1. In evaluation mode, normalise with those running values instead of the batch's.

Training mode: use batch statistics and update the running averages. Eval mode: use the stored running averages and change nothing.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
BatchNorm (gamma, beta)
normalise each feature over the batch, then rescale with learned gamma and beta
Your turn

Write the class BatchNorm(n, momentum=0.1, eps=1e-5) with gamma (ones), beta (zeros), running_mean (zeros), running_var (ones), a training flag (start True) and __call__(X) behaving as described. For running_var, the batch variance as np.var computes it (dividing by n) is the course's choice; PyTorch keeps the unbiased one (dividing by n - 1), and both are accepted.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.