What BatchNorm does to the numbers
Why it matters BatchNorm made very deep networks trainable in 2015. See what it does to the numbers before building it.
See it
Play the animation. A layer's outputs arrive with their own mean and spread per feature, some huge, some tiny, some shifted. BatchNorm takes each feature, over the current batch, subtracts its mean and divides by its spread. Every feature now has mean 0 and spread 1. Then it multiplies by a learned gamma and adds a learned beta, so the network can choose a different scale if that helps.
An honest note: why BatchNorm helps is still debated. The original paper's explanation ("internal covariate shift") is no longer the accepted one; smoother loss landscapes and allowing larger learning rates are better-supported reasons. What it does to the numbers is not in doubt.
BatchNorm: normalise each feature using the statistics of the current batch, then rescale with learned gamma and beta.
Words and symbols in this lesson
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
- BatchNorm (gamma, beta)
- normalise each feature over the batch, then rescale with learned gamma and beta
Predict before computing: a feature has values [2, 4, 6, 8] in a batch, gamma is 3 and beta is 1. Set my_output to the four numbers BatchNorm produces (use eps = 0). Then write batchnorm_forward(X, gamma, beta, eps=1e-5) for a 2D batch.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.