Sequences and state
Why it matters Text, speech, prices and sensor readings arrive one step at a time, and what came before changes what comes next. A recurrent network carries a running summary forward.
A recurrent neural network (RNN) keeps a hidden state h, a short vector. At every step it mixes the new input with the old state and squashes the result:
h = tanh(x_t @ Wx + h @ Wh + b)
The same weights are used at every step, just as a CNN reuses its filter at every position. The final h depends on everything the network has read, so it is a (lossy) memory of the whole sequence.
Tiny numbers
One input number, one hidden number, Wx = 1, Wh = 0.5, b = 0, h starts at 0. Reading 1 then 0: h = tanh(1) = 0.76, then h = tanh(0 + 0.38) = 0.36. The first input still shows in the state after the second step, fading.
Hidden state: the vector an RNN carries from step to step. Recurrent: feeding its own output back in as input.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
Write rnn_forward(xs, Wx, Wh, b): xs has shape (steps, input size); start from a zero state and return the list of hidden states after each step.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.