Syllabus Lesson 133 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

Gate: a broken RNN with two faults (AI off)

Why it matters Finding two faults at once, with no help, is what the clinics have been training you for.

AI tutor off. This character RNN has two faults. One keeps the loss from falling far, although every loss stays a finite number (nothing explodes). The other means the network never really uses what it read earlier in the snippet. Find and fix both.

Words and symbols in this lesson
recurrent network (RNN)
a network that reads one step at a time and carries a hidden state forward
entropy
how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Your turn

Fix init and run_snippet. After 200 steps, the average loss over the last 20 steps must be below 2.45, every loss must be a finite number, and the first character of a snippet must still affect the last prediction.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.