What the heads look at
Why it matters Attention weights can be read. Looking at what trained heads do is the start of interpretability: understanding a model from its insides, not only its outputs.
The recorded width-64 model has 2 layers of 4 heads. For each head, measure how much attention, on average, a position gives to the character just before it. If a head spread its attention evenly over everything it may see, that number would be about 0.10 over a 32-character snippet.
Predict: which layer's heads specialise in "look at the previous character"?
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- head
- one of H narrower attentions run side by side, each with its own pattern
Set my_guess to 0 or 1. Write prev_share(A) for one t by t attention matrix: the mean of A[i, i - 1] for i from 1 to t - 1. Fill share: (layer, head) to the average of prev_share over the 40 held-out snippets, using attention_maps.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.