Syllabus Lesson 148 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Attention has no sense of order

Why it matters "Dog bites man" and "man bites dog" use the same words. Self-attention, on its own, cannot tell them apart. Seeing why explains the extra ingredient every transformer adds.

Shuffle the rows of X and self-attention's output rows are shuffled the same way, nothing else changes. Each position's result depends on which vectors are present, not where they sit. (With the causal mask this is not quite true: position i sees exactly i + 1 tokens, so a model can learn to count them, and causal models with no position vectors do learn order that way. But it is slow and unreliable; explicit position vectors make order easy.)

So transformers add a position vector to each token's embedding before the first layer. The recorded GPT learned one vector per position, 32 of them. What happens to its predictions if we wipe them?

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
self-attention
every position attends to the positions of the same text
causal mask
scores above the diagonal set to minus infinity: no position sees the future
position vector
added to each token so attention can tell order
Your turn

Write shuffle_check(f): True if permuting the rows of a random X permutes f's output rows in the same way. Here f(X, Wq, Wk, Wv) returns just the output, like attn below; make random X and weights inside shuffle_check. Then fill with_pos and without_pos: the recorded model's held-out loss (model.eval_loss(held, n=32)) as it is and with model.P["pos"].data set to zeros (put it back afterwards).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.