Two networks in a game
Why it matters A VAE learns to rebuild pictures and pays for it with blur. A GAN (generative adversarial network) never rebuilds anything. One network makes pictures, a second one judges them, and each trains against the other. It is the idea behind the first generated faces that fooled people.
The two players:
- The generator G turns random noise z into a sample, G(z).
- The discriminator D looks at a sample and gives a score (a logit). sigmoid(score) is its belief that the sample is real.
D trains like any classifier: real samples labelled 1, generated ones labelled 0, binary cross-entropy. G trains to fool D. There are two ways to write G's loss, and the difference decides whether training starts at all:
- Saturating (the original paper's game): G minimises log(1 - D(G(z))).
- Non-saturating: G minimises -log D(G(z)). Same goal, "make D say real".
Tiny numbers: the slopes
Write f for D's score on a fake, so D(G(z)) = sigmoid(f). The slope of each loss with respect to f is:
saturating: -sigmoid(f) non-saturating: sigmoid(f) - 1
Early in training D easily spots fakes: D(G(z)) = 0.001. The saturating slope is then -0.001, almost nothing to learn from. The non-saturating slope is -0.999, a full push. That is the theory. The course measured when it matters: on the healthy ring both losses train (all 8 recorded seeds); in a setting where D wins, the saturating loss learns nothing.
All losses here use log-sum-exp tricks so no logarithm ever sees 0: log(sigmoid(f)) = -softplus(-f) and log(1 - sigmoid(f)) = -softplus(f), with softplus(a) = log(1 + e^a), computed as np.logaddexp(0, a).
Words and symbols in this lesson
- GAN
- generative adversarial network: a generator and a discriminator trained against each other
- generator (G)
- the network that turns noise into a sample
- discriminator (D)
- the network that scores a sample as real or generated
- saturating loss
- G minimises log(1 - D(G(z))); its slope is about 0 once D is sure, so G stops learning
Write the three losses as averages over a batch of scores: d_loss(real_scores, fake_scores), g_loss_saturating(fake_scores) and g_loss_nonsat(fake_scores). Then fill slopes: for each D(G(z)) in 0.001, 0.1 and 0.5, the slope of each generator loss with respect to the fake's score (for one sample).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.