Matching captions to pictures (the CLIP idea) (Optional)
Why it matters Text-to-image search, and the "understanding" part of image generators, start from one trick: train a picture encoder and a text encoder so that a picture and its caption land close together. It is InfoNCE again, across two kinds of data. This lesson is optional: it is not needed for the certificate.
Every photo here comes with a true caption, written from how it was generated: "a green ring on the left". The picture side is the frozen SimCLR encoder plus one trainable layer to 16 numbers. The text side is a bag of words (one number per word: is it in the caption?) times one trainable layer to 16 numbers.
The loss is symmetric: for a batch of n pairs, each picture must pick its caption out of the n captions, and each caption its picture out of the n pictures. Two cross-entropies over the same n x n similarity grid, one along the rows and one along the columns.
The test asks more than "right picture first": does the best-matching picture at least have the caption's colour? Its object? Its side?
Words and symbols in this lesson
- InfoNCE
- cross-entropy on a question the batch sets itself: which other view came from the same picture?
- temperature
- what similarities are divided by before the softmax; smaller makes the choice sharper
- SimCLR
- self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Write pair_loss(a, b, temperature) (engine tensors of shape (n, 16): normalise both with ssl.normalize (each row to length 1, as in the InfoNCE lesson), similarities divided by the temperature, cross-entropy along rows plus cross-entropy along columns) and train 600 steps of 128 pairs at temperature 0.1. Pass with the right picture first for at least 5% of test captions (chance 0.5%).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.