Syllabus Lesson 83 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

LayerNorm and RMSNorm

Why it matters Transformers do not use BatchNorm. They normalise each example on its own, which avoids the batch trap you just met and works for any batch size.

  • LayerNorm normalises across the features of each example (each row): subtract the row's mean, divide by the row's spread, then gamma and beta per feature. Same in training and use.
  • RMSNorm is simpler and used by many recent language models: skip the mean, and divide each row by its root mean square, sqrt(mean(x squared) + eps), then multiply by gamma.

Tiny numbers

Row [3, 4]. RMS = sqrt((9 + 16) / 2) = 3.536. RMSNorm gives [0.849, 1.131]. LayerNorm subtracts the mean 3.5 first: [-1, 1].

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
BatchNorm (gamma, beta)
normalise each feature over the batch, then rescale with learned gamma and beta
LayerNorm, RMSNorm
normalise each example on its own, across its features
Your turn

Write layernorm(X, gamma, beta, eps=1e-5) and rmsnorm(X, gamma, eps=1e-6), both per row.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.