Syllabus Lesson 149 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Positions as waves

Why it matters Learned position vectors stop at the longest text seen in training. The original transformer used fixed waves instead, which give every position a unique pattern and make "3 steps apart" look the same wherever it happens.

For position p and pair index i (0, 1, 2, ...), with d numbers in total:

PE[p, 2i]     = sin(p / 10000^(2i/d))
PE[p, 2i + 1] = cos(p / 10000^(2i/d))

The first pair is a fast wave, later pairs are slower and slower, like the hands of a clock: seconds, minutes, hours. Together they give each position a distinct reading.

The useful property: the dot product PE[p] . PE[p + k] depends only on k, the distance, not on p. (Each pair contributes cos of the angle difference, by the identity sin a sin b + cos a cos b = cos(a - b).)

Where the field is now Most open language models today use neither waves added to the input nor learned position vectors. RoPE (rotary position embedding, Su et al. 2021) rotates each query and key vector by an angle that grows with its position, after the query and key projections, so that the score between two positions depends only on how far apart they are. ALiBi (Press et al. 2021) adds no position vectors at all: it subtracts a penalty from each attention score that grows with distance. Both carry over better to texts longer than the training ones. The waves here are the original transformer's choice and the easiest to see.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
position vector
added to each token so attention can tell order
Your turn

Write positions(n, d) returning an n by d array. Then set same_distance to the largest difference between PE[p] . PE[p + 5] over p = 0 to 50 (it should be essentially 0).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.