Syllabus Lesson 158 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Gate: a broken transformer (AI off)

Why it matters This is a whole GPT forward pass written with the house engine, and it has two faults. Both are classic, and each one alone has fooled real teams.

AI tutor off. One fault lets the model see characters it is supposed to predict. The other makes the stack much harder to train. Find and fix both.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
head
one of H narrower attentions run side by side, each with its own pattern
Your turn

Fix logits. Changing a character must not change any prediction for earlier positions, and after 200 steps the average loss over the last 20 steps must be below 2.65.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.