LSTM gates, built
Why it matters The LSTM (1997, with the forget gate taught here added in 2000) fixed the RNN's memory problem and powered speech recognition and translation for twenty years. Its trick is simple once you see it.
See it
Play the animation. Besides the hidden state h, an LSTM keeps a cell c, a conveyor belt of memory. Three gates, each a sigmoid between 0 and 1, decide what happens to it:
- forget gate f: how much of the old cell to keep;
- input gate i: how much new information g to write in;
- output gate o: how much of the cell to show as h.
z = [x, h] @ W + b # one matrix for all four parts
i, f, g, o = sigmoid, sigmoid, tanh, sigmoid of the four quarters of z
c = f * c + i * g
h = o * tanh(c)
The key line is c = f * c + i * g. Going back one step, the gradient on the cell is multiplied by f, and nothing else: no repeated multiplication by a weight matrix. It still shrinks by f each step, so after k steps it is f to the power k. With f = 0.95 that is 0.08 after 50 steps; with f = 0.999 it is 0.95. So an LSTM remembers far when its forget gates stay very close to 1, and the next lesson measures how close is close enough.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- hidden state h
- the short vector of numbers an RNN carries from step to step
- LSTM
- an RNN with a memory cell c and three gates: forget, input, output
Write lstm_step(x, h, c, W, b) returning (h, c), with W of shape (inputs + hidden, 4 * hidden) and the four quarters in the order i, f, g, o. The inputs are batched: x is (batch, inputs), h and c are (batch, hidden), so join x and h along axis 1.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.