Syllabus Lesson 155 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

What the heads look at

Why it matters Attention weights can be read. Looking at what trained heads do is the start of interpretability: understanding a model from its insides, not only its outputs.

The recorded width-64 model has 2 layers of 4 heads. For each head, measure how much attention, on average, a position gives to the character just before it. If a head spread its attention evenly over everything it may see, that number would be about 0.10 over a 32-character snippet.

Predict: which layer's heads specialise in "look at the previous character"?

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
head
one of H narrower attentions run side by side, each with its own pattern
Your turn

Set my_guess to 0 or 1. Write prev_share(A) for one t by t attention matrix: the mean of A[i, i - 1] for i from 1 to t - 1. Fill share: (layer, head) to the average of prev_share over the 40 held-out snippets, using attention_maps.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.