Syllabus Lesson 79 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

What BatchNorm does to the numbers

Why it matters BatchNorm made very deep networks trainable in 2015. See what it does to the numbers before building it.

See it

Play the animation. A layer's outputs arrive with their own mean and spread per feature, some huge, some tiny, some shifted. BatchNorm takes each feature, over the current batch, subtracts its mean and divides by its spread. Every feature now has mean 0 and spread 1. Then it multiplies by a learned gamma and adds a learned beta, so the network can choose a different scale if that helps.

An honest note: why BatchNorm helps is still debated. The original paper's explanation ("internal covariate shift") is no longer the accepted one; smoother loss landscapes and allowing larger learning rates are better-supported reasons. What it does to the numbers is not in doubt.

BatchNorm: normalise each feature using the statistics of the current batch, then rescale with learned gamma and beta.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
BatchNorm (gamma, beta)
normalise each feature over the batch, then rescale with learned gamma and beta
Your turn

Predict before computing: a feature has values [2, 4, 6, 8] in a batch, gamma is 3 and beta is 1. Set my_output to the four numbers BatchNorm produces (use eps = 0). Then write batchnorm_forward(X, gamma, beta, eps=1e-5) for a 2D batch.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.