Where the parameters are
Why it matters "A 7 billion parameter model" is just this arithmetic at a bigger d. Being able to count tells you where memory goes and why width is expensive.
Per block, the big matrices are Wq, Wk, Wv and Wo (four d by d: 4d squared) and the feed-forward (d by 4d and 4d by d: 8d squared). That is 12 d squared, plus a few vectors of length d for the layer norms.
Outside the blocks: the token embeddings (V by d, also reused as the output layer in this model), the position vectors (ctx by d) and the final layer norm.
Doubling d multiplies the block parameters by 4, but not the time by 4. The recorded width-128 runs took about 2.3 times as long as width 64 on average (anywhere from 1.1 to 4 times from run to run: wall-clock time on a shared machine is noisy), and width 32 was no faster than width 64. In models this small, fixed costs per step (Python, small arrays) are a large part of the time; the 4x only shows once the matrix multiplies dominate.
Words and symbols in this lesson
- head
- one of H narrower attentions run side by side, each with its own pattern
- position vector
- added to each token so attention can tell order
- layer norm
- normalise each vector on its own: mean 0, spread 1, then times g plus b
- 12 d squared
- the parameters in one block's matrices
Write count(V, d, layers, ctx) that returns the exact parameter count of this GPT (each block has 12 d squared + 4d: two layer norms with g and b). It must match the real models.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.