A character RNN
Why it matters Predicting the next character from the ones before it is the original language model. The same idea, scaled up, is a chatbot.
Feed the course's own lesson text one character at a time. At each step the RNN updates its state and predicts the next character with a linear layer and softmax. The loss is the average cross-entropy over the 16 positions of each training snippet.
Each character goes in as an embedding: a learned row of numbers looked up by the character's index (Module 12 explains them properly; for now, a lookup table the network trains).
Call the Curve with the chart: guess the loss at step 0 and after 200 steps. A uniform guess over all the characters in the text gives about ln(V).
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- entropy
- how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Fill my_guess. Then write step_state(x_ids, h, P) with the house engine: tanh(embed(P["emb"], x_ids) + h @ P["Wh"] + P["bh"]). The training loop around it is written for you. After 200 steps the average loss over the last 20 steps must be below 2.6.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.