Syllabus Lesson 176 of 246 · Module 16: Two networks in a game (GANs, optional)
Module 16: Two networks in a game (GANs, optional)

Two networks in a game

Why it matters A VAE learns to rebuild pictures and pays for it with blur. A GAN (generative adversarial network) never rebuilds anything. One network makes pictures, a second one judges them, and each trains against the other. It is the idea behind the first generated faces that fooled people.

The two players:

  • The generator G turns random noise z into a sample, G(z).
  • The discriminator D looks at a sample and gives a score (a logit). sigmoid(score) is its belief that the sample is real.

D trains like any classifier: real samples labelled 1, generated ones labelled 0, binary cross-entropy. G trains to fool D. There are two ways to write G's loss, and the difference decides whether training starts at all:

  • Saturating (the original paper's game): G minimises log(1 - D(G(z))).
  • Non-saturating: G minimises -log D(G(z)). Same goal, "make D say real".

Tiny numbers: the slopes

Write f for D's score on a fake, so D(G(z)) = sigmoid(f). The slope of each loss with respect to f is:

saturating: -sigmoid(f)    non-saturating: sigmoid(f) - 1

Early in training D easily spots fakes: D(G(z)) = 0.001. The saturating slope is then -0.001, almost nothing to learn from. The non-saturating slope is -0.999, a full push. That is the theory. The course measured when it matters: on the healthy ring both losses train (all 8 recorded seeds); in a setting where D wins, the saturating loss learns nothing.

All losses here use log-sum-exp tricks so no logarithm ever sees 0: log(sigmoid(f)) = -softplus(-f) and log(1 - sigmoid(f)) = -softplus(f), with softplus(a) = log(1 + e^a), computed as np.logaddexp(0, a).

Words and symbols in this lesson
GAN
generative adversarial network: a generator and a discriminator trained against each other
generator (G)
the network that turns noise into a sample
discriminator (D)
the network that scores a sample as real or generated
saturating loss
G minimises log(1 - D(G(z))); its slope is about 0 once D is sure, so G stops learning
Your turn

Write the three losses as averages over a batch of scores: d_loss(real_scores, fake_scores), g_loss_saturating(fake_scores) and g_loss_nonsat(fake_scores). Then fill slopes: for each D(G(z)) in 0.001, 0.1 and 0.5, the slope of each generator loss with respect to the fake's score (for one sample).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.