One reads, one writes: the bottleneck
Why it matters Translation used to work like this: one RNN reads the sentence into a single vector, another writes the output from it. That one vector is a bottleneck, and seeing it fail is the reason attention was invented.
An encoder-decoder: the encoder RNN reads the input; its final hidden state is handed to the decoder RNN, which writes the output one symbol at a time, feeding each answer back in. Everything the decoder knows about the input must squeeze through that one vector of 48 numbers.
The recorded experiment seq2seq-bottleneck trained such networks to write a sequence of digits backwards, for lengths 4, 8, 12 and 16, two seeds each. It also trained a version with attention, which you will meet in Module 13. For now, look only at the plain ones.
Call the Curve: predict the digit accuracy of the plain model at length 16. Chance is 10%.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- encoder-decoder
- one RNN reads the input into a vector, another writes the output from it
- bottleneck
- the single vector everything must squeeze through
Set my_guess_16, then fill plain_acc: length to the mean digit accuracy of the two plain runs.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.