Gate: a broken transformer (AI off)
Why it matters This is a whole GPT forward pass written with the house engine, and it has two faults. Both are classic, and each one alone has fooled real teams.
AI tutor off. One fault lets the model see characters it is supposed to predict. The other makes the stack much harder to train. Find and fix both.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- head
- one of H narrower attentions run side by side, each with its own pattern
Fix logits. Changing a character must not change any prediction for earlier positions, and after 200 steps the average loss over the last 20 steps must be below 2.65.
AI off: no tutor, hints or worked solution on this one. Your own thinking only.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.