Attention fixes the bottleneck
Why it matters Attention was invented in 2014 to fix exactly the problem you measured in Module 11: an encoder squeezing a whole sentence into one vector.
In the recorded seq2seq-bottleneck experiment, the attention model's decoder, at every step, attends over all the encoder's hidden states (its query is the decoder's own state). It no longer depends on one final vector; it can look straight back at the digit it needs.
Predict: at length 16, where the plain model got about 59% of digits right, what does the attention model get?
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
Set my_guess to a number between 0 and 1: your guess for the attention model's digit accuracy at length 16. Fill acc: (model, length) to the mean digit accuracy of the two seeds, for both models and all four lengths, and gain_at_16.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.