Syllabus Lesson 145 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

The causal mask: no peeking

Why it matters A model trained to predict the next character must not see the next character. Without a mask it can simply copy the answer, look great in training, and then fail when generating, because at generation time the future does not exist yet.

See it

Play the animation. Position i may attend only to positions 0 to i. Before the softmax, every score above the diagonal is set to minus infinity, so its weight becomes exactly 0. The weights then form a lower triangle.

The test for a leak is direct: change a character at position j, and nothing the model computes for positions before j may change.

When the course trained a width-32 GPT for 1,200 steps without the mask, its training loss fell to 0.10, against 1.51 with the mask: it had learned to copy the answer from the next position, almost perfectly. Measured honestly, on unseen text and without letting it look ahead, its loss was 8.6 against 2.3. The lower number was a lie: the model was far worse. (Early on, at 400 steps, the two runs still looked alike, 2.28 against 2.24. Copying takes a while to discover, and then it takes over.)

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
causal mask
scores above the diagonal set to minus infinity: no position sees the future
head
one of H narrower attentions run side by side, each with its own pattern
Your turn

Write causal_attention(X, Wq, Wk, Wv) like lesson 4 but masked. Then leaks(f): return True if changing the last row of a random X changes any earlier row of f's output. Here f is any attention function called like yours, f(X, Wq, Wk, Wv), returning (output, weights); make random X and weights inside leaks.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.