Syllabus Lesson 151 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

One transformer block, by hand

Why it matters A GPT is the same block stacked. Write one with plain NumPy, run it with the recorded model's real weights, and if your numbers match the model's own, you understand every line of a transformer.

See it

Play the animation. A block takes h (t positions by d) and does two things, each added back onto h (the residual path of Module 6):

h = h + Wo-mixed multi-head causal attention of layer_norm(h)
h = h + feed-forward of layer_norm(h)        # relu(x @ W1) @ W2, W1 widens d to 4d

The recorded model's names: qkv{l} holds Wq, Wk and Wv side by side (d by 3d), o{l} is Wo, g1{l}, b1{l} and g2{l}, b2{l} are the two layer norms, w1{l}, w2{l} the feed-forward. Head h uses columns h times 16 to (h + 1) times 16 of each of Q, K and V.

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
head
one of H narrower attentions run side by side, each with its own pattern
layer norm
normalise each vector on its own: mean 0, spread 1, then times g plus b
transformer block
h + attention(norm(h)), then h + feed-forward(norm(h))
Your turn

Write block(h, l) for one sequence (h is t by d). The full model is assembled for you; its logits must match model.logits.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.