Attention has no sense of order
Why it matters "Dog bites man" and "man bites dog" use the same words. Self-attention, on its own, cannot tell them apart. Seeing why explains the extra ingredient every transformer adds.
Shuffle the rows of X and self-attention's output rows are shuffled the same way, nothing else changes. Each position's result depends on which vectors are present, not where they sit. (With the causal mask this is not quite true: position i sees exactly i + 1 tokens, so a model can learn to count them, and causal models with no position vectors do learn order that way. But it is slow and unreliable; explicit position vectors make order easy.)
So transformers add a position vector to each token's embedding before the first layer. The recorded GPT learned one vector per position, 32 of them. What happens to its predictions if we wipe them?
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- self-attention
- every position attends to the positions of the same text
- causal mask
- scores above the diagonal set to minus infinity: no position sees the future
- position vector
- added to each token so attention can tell order
Write shuffle_check(f): True if permuting the rows of a random X permutes f's output rows in the same way. Here f(X, Wq, Wk, Wv) returns just the output, like attn below; make random X and weights inside shuffle_check. Then fill with_pos and without_pos: the recorded model's held-out loss (model.eval_loss(held, n=32)) as it is and with model.P["pos"].data set to zeros (put it back afterwards).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.