The scale that keeps signals steady (Xavier and He)
Why it matters There is an exact answer to "how big should the starting weights be". Use it and deep networks become trainable.
Tiny numbers
One output of a layer adds up n_in products of an input and a weight. If inputs have spread 1 and weights have spread s, the sum of n_in such terms has spread s times sqrt(n_in) (variances add: Module 2, "How spread adds up"). To keep it at 1, choose s = 1 / sqrt(n_in). That is LeCun initialisation. Xavier (also called Glorot) initialisation also keeps the gradients steady on the way back, which depend on n_out, so it splits the difference: s = sqrt(2 / (n_in + n_out)). When n_in equals n_out the two are the same. Both suit tanh.
ReLU throws away the negative half, which halves the average squared size. Make up for it with twice the variance: s = sqrt(2 / n_in). That is He initialisation, right for ReLU. With width 100, He gives sqrt(0.02) = 0.141: exactly the "ok" scale from the last lesson.
Recap tanh: Xavier, sqrt(2 / (n_in + n_out)), or LeCun, 1 / sqrt(n_in). ReLU: He, sqrt(2 / n_in).
Words and symbols in this lesson
- initialisation
- how the weights are set before training starts
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Write xavier(n_in, n_out, rng) and he(n_in, n_out, rng) returning weight matrices. With He weights, the activation spread after 20 ReLU layers of width 256 must stay between 0.2 and 5.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.