Syllabus Lesson 152 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Where the parameters are

Why it matters "A 7 billion parameter model" is just this arithmetic at a bigger d. Being able to count tells you where memory goes and why width is expensive.

Per block, the big matrices are Wq, Wk, Wv and Wo (four d by d: 4d squared) and the feed-forward (d by 4d and 4d by d: 8d squared). That is 12 d squared, plus a few vectors of length d for the layer norms.

Outside the blocks: the token embeddings (V by d, also reused as the output layer in this model), the position vectors (ctx by d) and the final layer norm.

Doubling d multiplies the block parameters by 4, but not the time by 4. The recorded width-128 runs took about 2.3 times as long as width 64 on average (anywhere from 1.1 to 4 times from run to run: wall-clock time on a shared machine is noisy), and width 32 was no faster than width 64. In models this small, fixed costs per step (Python, small arrays) are a large part of the time; the 4x only shows once the matrix multiplies dominate.

Words and symbols in this lesson
head
one of H narrower attentions run side by side, each with its own pattern
position vector
added to each token so attention can tell order
layer norm
normalise each vector on its own: mean 0, spread 1, then times g plus b
12 d squared
the parameters in one block's matrices
Your turn

Write count(V, d, layers, ctx) that returns the exact parameter count of this GPT (each block has 12 d squared + 4d: two layer norms with g and b). It must match the real models.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.