Sampling text with temperature
Why it matters Every chatbot has a "temperature" setting. It is one division, and it trades safe, repetitive text for varied, riskier text.
To pick the next character, a model turns its scores into probabilities with softmax. Dividing the scores by a temperature T first changes how peaked that distribution is: T below 1 sharpens it (the favourite wins more often), T above 1 flattens it (surprises become likely).
The entropy, minus the sum of p log p, measures how spread out a distribution is: 0 when one choice is certain, ln(n) when all n are equally likely.
You will measure it on a real recorded model: the course's tiny GPT trained on its own lesson text.
Words and symbols in this lesson
- temperature
- divide the scores before softmax: below 1 sharper, above 1 flatter
- entropy
- how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Set my_order to the temperatures 0.5, 1.0 and 2.0 sorted from lowest to highest entropy (your prediction). Then write entropy_at(scores, T) for one row of scores, and fill measured with the average entropy of the model's next-character distribution over the given contexts, for each temperature.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.