One transformer block, by hand
Why it matters A GPT is the same block stacked. Write one with plain NumPy, run it with the recorded model's real weights, and if your numbers match the model's own, you understand every line of a transformer.
See it
Play the animation. A block takes h (t positions by d) and does two things, each added back onto h (the residual path of Module 6):
h = h + Wo-mixed multi-head causal attention of layer_norm(h)
h = h + feed-forward of layer_norm(h) # relu(x @ W1) @ W2, W1 widens d to 4d
The recorded model's names: qkv{l} holds Wq, Wk and Wv side by side (d by 3d), o{l} is Wo, g1{l}, b1{l} and g2{l}, b2{l} are the two layer norms, w1{l}, w2{l} the feed-forward. Head h uses columns h times 16 to (h + 1) times 16 of each of Q, K and V.
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- head
- one of H narrower attentions run side by side, each with its own pattern
- layer norm
- normalise each vector on its own: mean 0, spread 1, then times g plus b
- transformer block
- h + attention(norm(h)), then h + feed-forward(norm(h))
Write block(h, l) for one sequence (h is t by d). The full model is assembled for you; its logits must match model.logits.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.