Syllabus Lesson 81 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

BatchNorm backward on a batch of 3 numbers (optional)

Why it matters This is the hardest backward rule in Part 1, and optional. Its lesson: when every output depends on every input (through the mean and the spread), the gradient picks up extra terms.

Work it on one feature with a batch of 3 first. With x_hat = (x - mean) / spread and y = gamma x_hat + beta, the gradient with respect to x is not just gamma / spread times the arriving gradient. Nudging one x moves the mean and the spread, which moves every other output too. The result, per feature, with g the arriving gradient and N the batch size:

dx = (gamma / spread) * (g - g.mean() - x_hat * (g * x_hat).mean())

Read it as: the plain gradient, minus the part that only shifted everything together (the mean term), minus the part that only stretched everything together (the spread term). Those two parts are cancelled out by the normalisation itself.

A worked solution opens from the hints if you get stuck. Skipping this lesson does not block anything.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
BatchNorm (gamma, beta)
normalise each feature over the batch, then rescale with learned gamma and beta
Your turn

Write batchnorm_backward(X, gamma, g, eps=1e-5) returning (dX, dgamma, dbeta) for a 2D batch, where g is the gradient arriving at the output.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.