Syllabus Lesson 195 of 246 · Module 18: Learning without labels
Module 18: Learning without labels

Testing features with a simple classifier (linear probes)

Why it matters Nearest neighbours gave a rough answer. The standard test of learned features is a linear probe: freeze the features, train only a linear classifier on top with a few labels, and measure it on new photos. If a straight line can separate the objects, the features did the hard work.

The probe is the softmax regression of Module 3, on features instead of pixels:

  1. Standardise each feature with the training set's mean and spread (and use the same numbers on the test set).
  2. Gradient descent on cross-entropy, all training rows at once: 1000 steps, step size 0.5, weight decay 0.001.
  3. Accuracy on the test photos.

It is deliberately weak: a linear model with 400 labels. That is the point. It cannot invent good features; it can only read what the features already say.

Words and symbols in this lesson
linear probe
a linear classifier trained on frozen features: a test of what the features already say
SimCLR
self-supervised training with two views of each picture and InfoNCE (Chen et al. 2020)
Your turn

Write my_probe(F_train, y_train, F_test, y_test) as described (weights and biases start at zero). Then fill accuracy for a random encoder (ssl.Encoder(32, seed=0)) and the recorded SimCLR encoder with your probe. The raw-pixel baseline (768 numbers per photo) is computed for you with the course's probe, which does the same arithmetic in float32 on the fast matrix multiply: your float64 version would take half a minute on 768 inputs in the browser.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.