Syllabus Lesson 150 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Layer norm: keep each vector in range

Why it matters Module 6 kept activations healthy with batch norm, which averages over the batch. Text batches vary in length and generation runs one sequence at a time, so transformers normalise each position's vector on its own: layer norm.

For one vector x of d numbers: subtract its mean, divide by its standard deviation (plus a tiny epsilon, 1e-5, so we never divide by 0), then scale and shift by learned g and b:

y = (x - mean(x)) / sqrt(var(x) + 1e-5) * g + b

Every token's vector enters each attention and each feed-forward part with mean 0 and spread 1, however big the running sum has grown.

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
layer norm
normalise each vector on its own: mean 0, spread 1, then times g plus b
Your turn

Write layer_norm(x, g, b) for an array of any shape, normalising along the last axis. It must agree with the house engine's layernorm.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.