Syllabus Lesson 126 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

A character RNN

Why it matters Predicting the next character from the ones before it is the original language model. The same idea, scaled up, is a chatbot.

Feed the course's own lesson text one character at a time. At each step the RNN updates its state and predicts the next character with a linear layer and softmax. The loss is the average cross-entropy over the 16 positions of each training snippet.

Each character goes in as an embedding: a learned row of numbers looked up by the character's index (Module 12 explains them properly; for now, a lookup table the network trains).

Call the Curve with the chart: guess the loss at step 0 and after 200 steps. A uniform guess over all the characters in the text gives about ln(V).

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
entropy
how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Your turn

Fill my_guess. Then write step_state(x_ids, h, P) with the house engine: tanh(embed(P["emb"], x_ids) + h @ P["Wh"] + P["bh"]). The training loop around it is written for you. After 200 steps the average loss over the last 20 steps must be below 2.6.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.