Squeeze and rebuild (autoencoders)
Why it matters Every network so far turned a picture into a label. This module turns pictures into new pictures. The first step is a network that squeezes a picture into a few numbers and rebuilds it from them. Those few numbers are where new pictures will come from.
An autoencoder is two networks back to back. The encoder squeezes a 64-pixel digit into 16 numbers, its code. The decoder rebuilds 64 pixels from those 16 numbers. Training asks one thing: the rebuilt picture should match the original.
Nobody tells it what the 16 numbers should mean. It has to find a short description of a digit by itself, because 16 numbers cannot simply copy 64 pixels.
Tiny numbers
A 64-pixel digit becomes 16 numbers: a squeeze of 4 to 1. If the rebuilt pixels are off by 0.09 on average, the squared error per pixel is about 0.008. That is the size of error you will reach.
How the loss is measured
Each pixel is a number from 0 (blank) to 1 (full ink), so the decoder's last layer gives one score per pixel and the sigmoid turns it into 0..1. The loss is the binary cross-entropy from Module 3, summed over the 64 pixels and averaged over the batch. The house engine now has it as E.bce_with_logits(scores, pictures): it takes the scores before the sigmoid, so it never takes the log of 0.
The output also prints pictures as text: gen.show(pictures) prints each 8x8 digit in shades from blank to full ink.
Words and symbols in this lesson
- autoencoder
- an encoder that squeezes a picture into a few numbers and a decoder that rebuilds it
- code (latent)
- the few numbers the encoder makes from one picture
- mu, logvar
- the centre and the log of the squared spread of a code number in a VAE
Write encode(x): a dense layer with ReLU (P['e1']), then a dense layer to 16 numbers (P['e2']). Write decode(code) the same way with P['d1'], ReLU and P['d2'], returning the 64 scores before the sigmoid. The training loop is written for you. Reach a test rebuild error under 0.012 per pixel.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.