Too small, too big: vanishing and exploding
Why it matters A vanished signal means every gradient is nearly zero and training stalls. An exploded one means overflow and nan. Both come from the starting weights.
See it
Play the animation: the same 20-layer stack started three ways. Too small, and the signal fades layer by layer to nothing. Too big, and it doubles and doubles until it overflows. Just right, and it holds steady all the way through. The gradient on the way back suffers the same fate.
Vanishing: activations or gradients shrinking towards zero through the layers. Exploding: growing without limit.
Words and symbols in this lesson
- vanishing, exploding
- signals or gradients shrinking to nothing or growing without limit through layers
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Using your spreads from the last lesson (provided), write verdict(scale) returning "vanish" if the final spread is below 1e-3, "explode" if above 1e3, else "ok". Then fill in my_predictions for the scales 0.05, 0.14 and 0.5 before running.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.