The causal mask: no peeking
Why it matters A model trained to predict the next character must not see the next character. Without a mask it can simply copy the answer, look great in training, and then fail when generating, because at generation time the future does not exist yet.
See it
Play the animation. Position i may attend only to positions 0 to i. Before the softmax, every score above the diagonal is set to minus infinity, so its weight becomes exactly 0. The weights then form a lower triangle.
The test for a leak is direct: change a character at position j, and nothing the model computes for positions before j may change.
When the course trained a width-32 GPT for 1,200 steps without the mask, its training loss fell to 0.10, against 1.51 with the mask: it had learned to copy the answer from the next position, almost perfectly. Measured honestly, on unseen text and without letting it look ahead, its loss was 8.6 against 2.3. The lower number was a lie: the model was far worse. (Early on, at 400 steps, the two runs still looked alike, 2.28 against 2.24. Copying takes a while to discover, and then it takes over.)
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- causal mask
- scores above the diagonal set to minus infinity: no position sees the future
- head
- one of H narrower attentions run side by side, each with its own pattern
Write causal_attention(X, Wq, Wk, Wv) like lesson 4 but masked. Then leaks(f): return True if changing the last row of a random X changes any earlier row of f's output. Here f is any attention function called like yours, f(X, Wq, Wk, Wv), returning (output, weights); make random X and weights inside leaks.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.