Syllabus Lesson 131 of 246 · Module 11: Networks with memory (RNNs)
Module 11: Networks with memory (RNNs)

Sampling text with temperature

Why it matters Every chatbot has a "temperature" setting. It is one division, and it trades safe, repetitive text for varied, riskier text.

To pick the next character, a model turns its scores into probabilities with softmax. Dividing the scores by a temperature T first changes how peaked that distribution is: T below 1 sharpens it (the favourite wins more often), T above 1 flattens it (surprises become likely).

The entropy, minus the sum of p log p, measures how spread out a distribution is: 0 when one choice is certain, ln(n) when all n are equally likely.

You will measure it on a real recorded model: the course's tiny GPT trained on its own lesson text.

Words and symbols in this lesson
temperature
divide the scores before softmax: below 1 sharper, above 1 flatter
entropy
how spread out a distribution is: 0 when certain, ln(n) when all n choices are equal
Your turn

Set my_order to the temperatures 0.5, 1.0 and 2.0 sorted from lowest to highest entropy (your prediction). Then write entropy_at(scores, T) for one row of scores, and fill measured with the average entropy of the model's next-character distribution over the given contexts, for each temperature.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.