Positions as waves
Why it matters Learned position vectors stop at the longest text seen in training. The original transformer used fixed waves instead, which give every position a unique pattern and make "3 steps apart" look the same wherever it happens.
For position p and pair index i (0, 1, 2, ...), with d numbers in total:
PE[p, 2i] = sin(p / 10000^(2i/d))
PE[p, 2i + 1] = cos(p / 10000^(2i/d))
The first pair is a fast wave, later pairs are slower and slower, like the hands of a clock: seconds, minutes, hours. Together they give each position a distinct reading.
The useful property: the dot product PE[p] . PE[p + k] depends only on k, the distance, not on p. (Each pair contributes cos of the angle difference, by the identity sin a sin b + cos a cos b = cos(a - b).)
Where the field is now Most open language models today use neither waves added to the input nor learned position vectors. RoPE (rotary position embedding, Su et al. 2021) rotates each query and key vector by an angle that grows with its position, after the query and key projections, so that the score between two positions depends only on how far apart they are. ALiBi (Press et al. 2021) adds no position vectors at all: it subtracts a penalty from each attention score that grows with distance. Both carry over better to texts longer than the training ones. The waves here are the original transformer's choice and the easiest to see.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
- position vector
- added to each token so attention can tell order
Write positions(n, d) returning an n by d array. Then set same_distance to the largest difference between PE[p] . PE[p + 5] over p = 0 to 50 (it should be essentially 0).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.