Who remembers? The recorded runs
Why it matters The textbook says LSTMs remember and plain RNNs forget. The course tested that, and the real answer has a twist worth knowing before you ever train one.
The recorded experiment rnn-vs-lstm-memory: the network sees one of four symbols, then a gap of 5, 15, 30 or 60 more symbols drawn from the same alphabet, and must say what the first symbol was. Three kinds of network, three seeds each:
- a plain tanh RNN;
- an LSTM whose forget gate starts with bias 1 (it keeps about 73% of the cell per step at first);
- an LSTM whose forget gate starts with bias 3 (about 95%).
Before you read the runs, predict: which kind succeeds on all three seeds at a gap of 60?
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- LSTM
- an RNN with a memory cell c and three gates: forget, input, output
Set my_prediction to "rnn", "lstm-forget-bias-1" or "lstm-forget-bias-3". Then fill successes: (cell, gap) to how many of the three seeds reached at least 95% test accuracy, using nnsim.recorded.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.