Syllabus Lesson 129 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

LSTM gates, built

Why it matters The LSTM (1997, with the forget gate taught here added in 2000) fixed the RNN's memory problem and powered speech recognition and translation for twenty years. Its trick is simple once you see it.

See it

Play the animation. Besides the hidden state h, an LSTM keeps a cell c, a conveyor belt of memory. Three gates, each a sigmoid between 0 and 1, decide what happens to it:

  • forget gate f: how much of the old cell to keep;
  • input gate i: how much new information g to write in;
  • output gate o: how much of the cell to show as h.
z = [x, h] @ W + b            # one matrix for all four parts
i, f, g, o = sigmoid, sigmoid, tanh, sigmoid of the four quarters of z
c = f * c + i * g
h = o * tanh(c)

The key line is c = f * c + i * g. Going back one step, the gradient on the cell is multiplied by f, and nothing else: no repeated multiplication by a weight matrix. It still shrinks by f each step, so after k steps it is f to the power k. With f = 0.95 that is 0.08 after 50 steps; with f = 0.999 it is 0.95. So an LSTM remembers far when its forget gates stay very close to 1, and the next lesson measures how close is close enough.

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
hidden state h
the short vector of numbers an RNN carries from step to step
LSTM
an RNN with a memory cell c and three gates: forget, input, output
Your turn

Write lstm_step(x, h, c, W, b) returning (h, c), with W of shape (inputs + hidden, 4 * hidden) and the four quarters in the order i, f, g, o. The inputs are batched: x is (batch, inputs), h and c are (batch, hidden), so join x and h along axis 1.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.