Syllabus Lesson 78 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

Dead ReLUs

Why it matters A ReLU that is off for every input passes no gradient, so it never learns again. Whole layers can quietly die.

A unit is dead if its output is zero for every example in a batch. Common causes: a large negative bias, or one huge update that pushed its weights far negative.

Tiny numbers

Activations for 4 examples and 3 units: [[0, 1.2, 0], [0, 0.3, 0], [0, 2.0, 0.1], [0, 0, 0]]. Unit 1 is zero everywhere: dead. Unit 3 fired once: alive. Dead fraction: 1 of 3.

Fixes: sensible starting weights, biases starting at 0 (or slightly positive), a smaller learning rate, or leaky ReLU, which keeps a small slope (say 0.01) for negative inputs so the unit can recover.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Your turn

Write dead_fraction(H) for an activation matrix (examples by units). Then, for the layer below, which was started with biases of -4, write fixed_layer(X, W) that keeps the same weights but uses biases of zero and a leaky ReLU with slope 0.01; its dead fraction (counting exactly-zero outputs) must be 0.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.