Pick your partner out of the batch (InfoNCE)
Why it matters SimCLR's whole training signal is one loss, and you already know it: it is cross-entropy, on a multiple-choice question the network sets itself.
Take a batch of n photos and make two random views of each (crop, flip, change brightness and colour). That gives 2n views. For each view, the question is: which of the other 2n - 1 views came from the same photo? Exactly one did.
- Encode every view to a short vector z, and scale each z to length 1.
- Similarity of two views = their dot product (the cosine), divided by a temperature (0.2 here): small temperatures make the scores sharper.
- A view may not pick itself: its own score is set to minus infinity.
- Cross-entropy, with the right answer being the partner: view i's partner is i + n (and the other way round).
Watch the scene: six photos, two views each, the grid of similarities, and each row's choice.
Tiny numbers
Two photos, so four views. If view 0's similarities to views 1, 2, 3 are 0.1, 0.9, 0.2 (its partner is view 2), divided by 0.2 they become 0.5, 4.5, 1.0. Softmax gives the partner 0.954, and view 0's loss is -log 0.954 = 0.047.
Words and symbols in this lesson
- view
- a randomly changed copy of a picture: cropped, flipped, brighter, less colourful
- InfoNCE
- cross-entropy on a question the batch sets itself: which other view came from the same picture?
- temperature
- what similarities are divided by before the softmax; smaller makes the choice sharper
- SimCLR
- self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Write info_nce(z, temperature) in plain NumPy for z of shape (2n, d), returning the mean loss over all 2n rows. Then compute example: the loss for the four-view batch in the starter.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.