Syllabus Lesson 132 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

One reads, one writes: the bottleneck

Why it matters Translation used to work like this: one RNN reads the sentence into a single vector, another writes the output from it. That one vector is a bottleneck, and seeing it fail is the reason attention was invented.

An encoder-decoder: the encoder RNN reads the input; its final hidden state is handed to the decoder RNN, which writes the output one symbol at a time, feeding each answer back in. Everything the decoder knows about the input must squeeze through that one vector of 48 numbers.

The recorded experiment seq2seq-bottleneck trained such networks to write a sequence of digits backwards, for lengths 4, 8, 12 and 16, two seeds each. It also trained a version with attention, which you will meet in Module 13. For now, look only at the plain ones.

Call the Curve: predict the digit accuracy of the plain model at length 16. Chance is 10%.

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
encoder-decoder
one RNN reads the input into a vector, another writes the output from it
bottleneck
the single vector everything must squeeze through
Your turn

Set my_guess_16, then fill plain_acc: length to the mean digit accuracy of the two plain runs.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.