LayerNorm and RMSNorm
Why it matters Transformers do not use BatchNorm. They normalise each example on its own, which avoids the batch trap you just met and works for any batch size.
- LayerNorm normalises across the features of each example (each row): subtract the row's mean, divide by the row's spread, then gamma and beta per feature. Same in training and use.
- RMSNorm is simpler and used by many recent language models: skip the mean, and divide each row by its root mean square,
sqrt(mean(x squared) + eps), then multiply by gamma.
Tiny numbers
Row [3, 4]. RMS = sqrt((9 + 16) / 2) = 3.536. RMSNorm gives [0.849, 1.131]. LayerNorm subtracts the mean 3.5 first: [-1, 1].
Words and symbols in this lesson
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
- BatchNorm (gamma, beta)
- normalise each feature over the batch, then rescale with learned gamma and beta
- LayerNorm, RMSNorm
- normalise each example on its own, across its features
Write layernorm(X, gamma, beta, eps=1e-5) and rmsnorm(X, gamma, eps=1e-6), both per row.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.