Train a tiny GPT, live
Why it matters Everything comes together: embeddings, positions, masked multi-head attention, residuals and layer norms, trained with Adam on next-character prediction. On your own CPU, in seconds.
Width 32, 2 blocks, 4 heads, 32 characters of context: about 29,000 parameters. 150 steps of 16 snippets each, a few seconds on a laptop. Call the Curve: the loss at the start and after 150 steps. (A uniform guess over 95 characters is ln 95 = 4.55.)
Then sample from it. After 150 steps it makes letter soup with spaces in roughly the right places; the recorded width-64 model, trained for 2,000 steps, writes word-shaped text. Same code, more compute.
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- head
- one of H narrower attentions run side by side, each with its own pattern
- layer norm
- normalise each vector on its own: mean 0, spread 1, then times g plus b
Fill my_guess, then train with model.train (150 steps, learning rate 0.003, batch 16) on corpus.TEXT and keep the history in hist. The call is hist = model.train(text, steps=..., lr=..., batch=...); it returns a dictionary whose hist['loss'] is the training loss logged every 20 steps (and at the last step), and model.generate(prompt, n, temperature) writes text afterwards. (help(model.train) prints this too.) The average of the last 5 logged losses must be below 2.85.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.