Syllabus Lesson 77 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

The scale that keeps signals steady (Xavier and He)

Why it matters There is an exact answer to "how big should the starting weights be". Use it and deep networks become trainable.

Tiny numbers

One output of a layer adds up n_in products of an input and a weight. If inputs have spread 1 and weights have spread s, the sum of n_in such terms has spread s times sqrt(n_in) (variances add: Module 2, "How spread adds up"). To keep it at 1, choose s = 1 / sqrt(n_in). That is LeCun initialisation. Xavier (also called Glorot) initialisation also keeps the gradients steady on the way back, which depend on n_out, so it splits the difference: s = sqrt(2 / (n_in + n_out)). When n_in equals n_out the two are the same. Both suit tanh.

ReLU throws away the negative half, which halves the average squared size. Make up for it with twice the variance: s = sqrt(2 / n_in). That is He initialisation, right for ReLU. With width 100, He gives sqrt(0.02) = 0.141: exactly the "ok" scale from the last lesson.

Recap tanh: Xavier, sqrt(2 / (n_in + n_out)), or LeCun, 1 / sqrt(n_in). ReLU: He, sqrt(2 / n_in).

Words and symbols in this lesson
initialisation
how the weights are set before training starts
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Your turn

Write xavier(n_in, n_out, rng) and he(n_in, n_out, rng) returning weight matrices. With He weights, the activation spread after 20 ReLU layers of width 256 must stay between 0.2 and 5.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.