Syllabus Lesson 75 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

Signals through 20 layers

Why it matters Before a deep network learns anything, the signal must survive the trip through every layer. Often it does not, and no optimiser can fix that.

Push a batch of random inputs through 20 layers of h = relu(h @ W), with every weight drawn from a normal distribution of some spread. After each layer, measure the spread (standard deviation) of the activations.

Tiny numbers

Each layer multiplies the typical size of the signal by roughly the same factor. If that factor is 0.5, after 20 layers the signal is 0.5 to the 20th: about one millionth. If it is 2, it is about a million times bigger. Only a factor near 1 survives.

Recap Depth turns small per-layer changes into huge ones.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Your turn

Write spreads(scale, depth=20, width=100, seed=0): start from h = rng.standard_normal((256, width)), then for each layer draw W = rng.normal(0, scale, (width, width)) and set h = relu(h @ W). Return the list of h.std() after each layer.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.