Syllabus Lesson 197 of 246 · Module 18: Learning without labels
Module 18: Learning without labels

SimCLR: two views of one photo

Why it matters Now the whole method: random views plus InfoNCE, repeated. No labels at any point. What it learns depends almost entirely on one choice you make: which differences between two views it is told to ignore.

Each step: take a batch of photos, make two random views of each with ssl.augment, encode both, project them with a small head, and minimise InfoNCE. The augmentations define the task: whatever differs between two views of one photo (position of the crop, mirror image, brightness, saturation) is what the network learns to ignore. Whatever stays the same (the object) is what it learns to keep.

Real SimCLR runs use thousands of steps and big batches. Here you run a short one live (width 8, batch 32, 80 steps) to watch the loss fall, then read the course's recorded long runs (width 32, 2500 steps of 128).

The recorded runs include two variations on the same recipe: adding a random hue rotation to the views (the SimCLR paper found strong colour changes essential on its photos), and a higher temperature (0.5).

Where the field is now SimCLR needs many negatives (other photos in the batch), so it trains best with very large batches (the paper used 4,096). Later methods drop the negatives: BYOL (2020) and DINO (2021) train a network to match a slowly updated copy of itself, and MAE (2021) hides three quarters of an image's patches and rebuilds them, the picture version of this module's fill-in-the-blank. For text, the same split exists: masked prediction (BERT) and next-token prediction (GPT).

Words and symbols in this lesson
masked prediction
hiding part of the input and training the network to fill it in
view
a randomly changed copy of a picture: cropped, flipped, brighter, less colourful
augmentation
the random change that makes a view; it decides what the network learns to ignore
InfoNCE
cross-entropy on a question the batch sets itself: which other view came from the same picture?
temperature
what similarities are divided by before the softmax; smaller makes the choice sharper
SimCLR
self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Your turn

Run the live pretraining and fill live with the mean loss over the first 10 and the last 10 steps. Then fill report from the recorded runs: the probe accuracy (original objects) for raw pixels, the random encoder, SimCLR, SimCLR with hue rotation, and supervised, plus gap, how far SimCLR is behind supervised, as a fraction (0.05 means 5 points). Finally set hue_helped to True or False.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.