Syllabus Lesson 76 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

Too small, too big: vanishing and exploding

Why it matters A vanished signal means every gradient is nearly zero and training stalls. An exploded one means overflow and nan. Both come from the starting weights.

See it

Play the animation: the same 20-layer stack started three ways. Too small, and the signal fades layer by layer to nothing. Too big, and it doubles and doubles until it overflows. Just right, and it holds steady all the way through. The gradient on the way back suffers the same fate.

Vanishing: activations or gradients shrinking towards zero through the layers. Exploding: growing without limit.

Words and symbols in this lesson
vanishing, exploding
signals or gradients shrinking to nothing or growing without limit through layers
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Your turn

Using your spreads from the last lesson (provided), write verdict(scale) returning "vanish" if the final spread is below 1e-3, "explode" if above 1e3, else "ok". Then fill in my_predictions for the scales 0.05, 0.14 and 0.5 before running.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.