Layer norm: keep each vector in range
Why it matters Module 6 kept activations healthy with batch norm, which averages over the batch. Text batches vary in length and generation runs one sequence at a time, so transformers normalise each position's vector on its own: layer norm.
For one vector x of d numbers: subtract its mean, divide by its standard deviation (plus a tiny epsilon, 1e-5, so we never divide by 0), then scale and shift by learned g and b:
y = (x - mean(x)) / sqrt(var(x) + 1e-5) * g + b
Every token's vector enters each attention and each feed-forward part with mean 0 and spread 1, however big the running sum has grown.
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- layer norm
- normalise each vector on its own: mean 0, spread 1, then times g plus b
Write layer_norm(x, g, b) for an array of any shape, normalising along the last axis. It must agree with the house engine's layernorm.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.