Syllabus Lesson 125 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

Sequences and state

Why it matters Text, speech, prices and sensor readings arrive one step at a time, and what came before changes what comes next. A recurrent network carries a running summary forward.

A recurrent neural network (RNN) keeps a hidden state h, a short vector. At every step it mixes the new input with the old state and squashes the result:

h = tanh(x_t @ Wx + h @ Wh + b)

The same weights are used at every step, just as a CNN reuses its filter at every position. The final h depends on everything the network has read, so it is a (lossy) memory of the whole sequence.

Tiny numbers

One input number, one hidden number, Wx = 1, Wh = 0.5, b = 0, h starts at 0. Reading 1 then 0: h = tanh(1) = 0.76, then h = tanh(0 + 0.38) = 0.36. The first input still shows in the state after the second step, fading.

Hidden state: the vector an RNN carries from step to step. Recurrent: feeding its own output back in as input.

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
Your turn

Write rnn_forward(xs, Wx, Wh, b): xs has shape (steps, input size); start from a zero state and return the list of hidden states after each step.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.