Syllabus Lesson 130 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

Who remembers? The recorded runs

Why it matters The textbook says LSTMs remember and plain RNNs forget. The course tested that, and the real answer has a twist worth knowing before you ever train one.

The recorded experiment rnn-vs-lstm-memory: the network sees one of four symbols, then a gap of 5, 15, 30 or 60 more symbols drawn from the same alphabet, and must say what the first symbol was. Three kinds of network, three seeds each:

  • a plain tanh RNN;
  • an LSTM whose forget gate starts with bias 1 (it keeps about 73% of the cell per step at first);
  • an LSTM whose forget gate starts with bias 3 (about 95%).

Before you read the runs, predict: which kind succeeds on all three seeds at a gap of 60?

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
LSTM
an RNN with a memory cell c and three gates: forget, input, output
Your turn

Set my_prediction to "rnn", "lstm-forget-bias-1" or "lstm-forget-bias-3". Then fill successes: (cell, gap) to how many of the three seeds reached at least 95% test accuracy, using nnsim.recorded.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.