Signals through 20 layers
Why it matters Before a deep network learns anything, the signal must survive the trip through every layer. Often it does not, and no optimiser can fix that.
Push a batch of random inputs through 20 layers of h = relu(h @ W), with every weight drawn from a normal distribution of some spread. After each layer, measure the spread (standard deviation) of the activations.
Tiny numbers
Each layer multiplies the typical size of the signal by roughly the same factor. If that factor is 0.5, after 20 layers the signal is 0.5 to the 20th: about one millionth. If it is 2, it is about a million times bigger. Only a factor near 1 survives.
Recap Depth turns small per-layer changes into huge ones.
Words and symbols in this lesson
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Write spreads(scale, depth=20, width=100, seed=0): start from h = rng.standard_normal((256, width)), then for each layer draw W = rng.normal(0, scale, (width, width)) and set h = relu(h @ W). Return the list of h.std() after each layer.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.