Dropout, built
Why it matters Dropout is a simple way to stop a network leaning too hard on any single unit, and you will see it inside many models.
During training, dropout switches off each unit at random with probability p (say 0.5) on every step. The network cannot rely on any one unit, because it might be gone next step, so it spreads what it knows across many.
Keeping the average right
If half the units are switched off, the next layer receives about half as much signal. Modern inverted dropout fixes this during training: the units that survive are multiplied by 1 / (1 - p), here 2. At evaluation time dropout does nothing at all, and the average signal matches training. (The original 2014 paper scaled at evaluation time instead; the result is the same, but inverted dropout keeps evaluation code untouched.)
Dropout rate p: the probability of switching a unit off during training.
Words and symbols in this lesson
- dropout (p)
- switching units off at random with probability p during training
Write dropout(h, p, training, rng): in training, zero each entry with probability p and scale survivors by 1 / (1 - p); otherwise return h unchanged. Use rng.random(h.shape) >= p as the keep mask.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.