SimCLR: two views of one photo
Why it matters Now the whole method: random views plus InfoNCE, repeated. No labels at any point. What it learns depends almost entirely on one choice you make: which differences between two views it is told to ignore.
Each step: take a batch of photos, make two random views of each with ssl.augment, encode both, project them with a small head, and minimise InfoNCE. The augmentations define the task: whatever differs between two views of one photo (position of the crop, mirror image, brightness, saturation) is what the network learns to ignore. Whatever stays the same (the object) is what it learns to keep.
Real SimCLR runs use thousands of steps and big batches. Here you run a short one live (width 8, batch 32, 80 steps) to watch the loss fall, then read the course's recorded long runs (width 32, 2500 steps of 128).
The recorded runs include two variations on the same recipe: adding a random hue rotation to the views (the SimCLR paper found strong colour changes essential on its photos), and a higher temperature (0.5).
Where the field is now SimCLR needs many negatives (other photos in the batch), so it trains best with very large batches (the paper used 4,096). Later methods drop the negatives: BYOL (2020) and DINO (2021) train a network to match a slowly updated copy of itself, and MAE (2021) hides three quarters of an image's patches and rebuilds them, the picture version of this module's fill-in-the-blank. For text, the same split exists: masked prediction (BERT) and next-token prediction (GPT).
Words and symbols in this lesson
- masked prediction
- hiding part of the input and training the network to fill it in
- view
- a randomly changed copy of a picture: cropped, flipped, brighter, less colourful
- augmentation
- the random change that makes a view; it decides what the network learns to ignore
- InfoNCE
- cross-entropy on a question the batch sets itself: which other view came from the same picture?
- temperature
- what similarities are divided by before the softmax; smaller makes the choice sharper
- SimCLR
- self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Run the live pretraining and fill live with the mean loss over the first 10 and the last 10 steps. Then fill report from the recorded runs: the probe accuracy (original objects) for raw pixels, the random encoder, SimCLR, SimCLR with hue rotation, and supervised, plus gap, how far SimCLR is behind supervised, as a fraction (0.05 means 5 points). Finally set hue_helped to True or False.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.