Dead ReLUs
Why it matters A ReLU that is off for every input passes no gradient, so it never learns again. Whole layers can quietly die.
A unit is dead if its output is zero for every example in a batch. Common causes: a large negative bias, or one huge update that pushed its weights far negative.
Tiny numbers
Activations for 4 examples and 3 units: [[0, 1.2, 0], [0, 0.3, 0], [0, 2.0, 0.1], [0, 0, 0]]. Unit 1 is zero everywhere: dead. Unit 3 fired once: alive. Dead fraction: 1 of 3.
Fixes: sensible starting weights, biases starting at 0 (or slightly positive), a smaller learning rate, or leaky ReLU, which keeps a small slope (say 0.01) for negative inputs so the unit can recover.
Words and symbols in this lesson
- Xavier, LeCun, He
- starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Write dead_fraction(H) for an activation matrix (examples by units). Then, for the layer below, which was started with biases of -4, write fixed_layer(X, W) that keeps the same weights but uses biases of zero and a leaky ReLU with slope 0.01; its dead fraction (counting exactly-zero outputs) must be 0.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.