Syllabus Lesson 118 of 246 · Module 10: Networks that see (CNNs)
Module 10: Networks that see (CNNs)

Fewer bits: float32, float16 and loss scaling

Why it matters Numbers stored in fewer bits are faster to move and take half the memory, so big models train and ship in 16-bit numbers. But 16 bits break in two ways you must know.

  • Overflow: float16 cannot hold anything above 65,504. A bigger value becomes infinity, and then nan spreads through training.
  • Underflow: float16 cannot hold tiny values either; anything smaller than about 3e-8 rounds to exactly 0. Small gradients vanish.

Loss scaling

Multiply the loss by a big number S (say 1,024) before the backward pass, so every gradient is S times bigger and stays above the underflow line. Convert to float16, then divide by S again in float32 before the update. The gradients survive.

The course's own recorded checkpoints are stored in float16 for exactly this reason: half the size.

Most large models today train in a third format, bfloat16: 16 bits like float16, but it keeps float32's range (up to about 3e38) and gives up precision instead (about 3 significant digits). Overflow and underflow mostly stop being problems, so loss scaling is rarely needed with it; the price is rounding, which is why the weights and the update are still kept in float32.

Words and symbols in this lesson
float16
16-bit numbers: half the memory, overflow above 65,504, tiny values become 0
RECORDED
a real run the course trained once and ships for you to study
Your turn

Write to_f16(g) (convert to float16 and back to float32) and scaled_to_f16(g, S) (multiply by S, convert, convert back, divide by S). Set lost_plain and lost_scaled to the fraction of the given small gradients that become exactly zero each way.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.