Why divide by the square root of d
Why it matters Every transformer divides its attention scores by the square root of the key size. Leave it out and attention stops blending and its gradients nearly vanish. You will measure why.
A dot product of two random vectors of length d adds up d products. Each has spread about 1, so the sum has spread about the square root of d. The reason: when you add independent random numbers, their spreads do not add, their squares do (the squares are called variances). d products, each with variance 1, give variance d, so the spread is the square root of d. For d = 64 that is 8.
Scores spread over plus and minus 8 or more make softmax almost one-hot: one weight near 1, the rest near 0. A one-hot softmax has tiny gradients: the slope of each output with respect to its own score is p(1 - p), which is near 0 when p is near 0 or 1 (Module 2's softmax-gradient lesson derives it, and this module's backward-through-attention lesson uses it), so the model learns slowly. Dividing by the square root of d brings the spread back to about 1, whatever d is.
Predict: as d grows from 4 to 256, what happens to the biggest weight, with and without the division?
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
Fill my_guess. Then write top_weight(d, scaled): 2,000 random queries against 32 random keys each (normal numbers, seed 0), the average of the biggest softmax weight. Fill measured.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.