Syllabus Lesson 196 of 246 · Module 18: Learning without labels
Module 18: Learning without labels

Pick your partner out of the batch (InfoNCE)

Why it matters SimCLR's whole training signal is one loss, and you already know it: it is cross-entropy, on a multiple-choice question the network sets itself.

Take a batch of n photos and make two random views of each (crop, flip, change brightness and colour). That gives 2n views. For each view, the question is: which of the other 2n - 1 views came from the same photo? Exactly one did.

  1. Encode every view to a short vector z, and scale each z to length 1.
  2. Similarity of two views = their dot product (the cosine), divided by a temperature (0.2 here): small temperatures make the scores sharper.
  3. A view may not pick itself: its own score is set to minus infinity.
  4. Cross-entropy, with the right answer being the partner: view i's partner is i + n (and the other way round).

Watch the scene: six photos, two views each, the grid of similarities, and each row's choice.

Tiny numbers

Two photos, so four views. If view 0's similarities to views 1, 2, 3 are 0.1, 0.9, 0.2 (its partner is view 2), divided by 0.2 they become 0.5, 4.5, 1.0. Softmax gives the partner 0.954, and view 0's loss is -log 0.954 = 0.047.

Words and symbols in this lesson
view
a randomly changed copy of a picture: cropped, flipped, brighter, less colourful
InfoNCE
cross-entropy on a question the batch sets itself: which other view came from the same picture?
temperature
what similarities are divided by before the softmax; smaller makes the choice sharper
SimCLR
self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Your turn

Write info_nce(z, temperature) in plain NumPy for z of shape (2n, d), returning the mean loss over all 2n rows. Then compute example: the loss for the four-view batch in the starter.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.