Syllabus Lesson 175 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

Gate: the VAE that draws one digit (AI off)

Why it matters This failure has a name, posterior collapse, and it is common enough that every generative model you build later can hit some version of it. Here you diagnose it and fix it alone.

Gate rules: AI tutor off. Run the starter and look at what the broken VAE makes. You may train as many variations as you like with gen.train_vae(...) and judge them with gen.judge, active_units() and the rebuild error.

Three settings exist, each with a reason to try it:

  • weight: the strength of pull 2 (the KL), 1 by default.
  • warmup_steps: start with weight 0 and raise it evenly to weight over this many steps (training is 600 steps).
  • free_bits: a floor per code number. A number whose KL is under the floor pays the floor and gets no push towards the centre, so the model can afford to use it.

You will read online that warming the weight up is the standard fix. Find out whether it is here.

The check retrains your fix on a new seed and passes it only if all four hold: test rebuild error under 0.035, at least 3 active code numbers, new digits with closeness at most 1.33 and variety at least 3.0.

Words and symbols in this lesson
code (latent)
the few numbers the encoder makes from one picture
VAE
variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
closeness, variety
the course judge: distance to the nearest real digit, and bits of spread over the ten digits
posterior collapse
a VAE whose codes all become N(0, 1) and carry nothing; it draws one average picture
Your turn

Set diagnosis to one letter and fix to a dict of settings for gen.train_vae that cures the collapse.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.