Gate: a broken RNN with two faults (AI off)
Why it matters Finding two faults at once, with no help, is what the clinics have been training you for.
AI tutor off. This character RNN has two faults. One keeps the loss from falling far, although every loss stays a finite number (nothing explodes). The other means the network never really uses what it read earlier in the snippet. Find and fix both.
Words and symbols in this lesson
- recurrent network (RNN)
- a network that reads one step at a time and carries a hidden state forward
- entropy
- how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Fix init and run_snippet. After 200 steps, the average loss over the last 20 steps must be below 2.45, every loss must be a finite number, and the first character of a snippet must still affect the last prediction.
AI off: no tutor, hints or worked solution on this one. Your own thinking only.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.