Syllabus Lesson 143 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Attention fixes the bottleneck

Why it matters Attention was invented in 2014 to fix exactly the problem you measured in Module 11: an encoder squeezing a whole sentence into one vector.

In the recorded seq2seq-bottleneck experiment, the attention model's decoder, at every step, attends over all the encoder's hidden states (its query is the decoder's own state). It no longer depends on one final vector; it can look straight back at the digit it needs.

Predict: at length 16, where the plain model got about 59% of digits right, what does the attention model get?

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
Your turn

Set my_guess to a number between 0 and 1: your guess for the attention model's digit accuracy at length 16. Fill acc: (model, length) to the mean digit accuracy of the two seeds, for both models and all four lengths, and gain_at_16.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.