Syllabus Lesson 215 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

Teaching preferences (DPO)

Why it matters Chat models are not only trained on text; they are tuned on preferences: of two answers, which one people liked better. DPO (Rafailov et al. 2023) does it directly from pairs, with one loss and no separate reward model. It is also easy to overdo.

The preference here is a style the base model never uses: joining words with underscores ("snake_case talk"). Each pair has a 20-character prompt from the lesson text, the real next 10 characters (rejected) and the same 10 with every space written as an underscore (chosen).

The DPO loss for one pair compares how much more likely the tuned model makes the chosen answer than the reference model did, against the same for the rejected one:

-log sigmoid( beta x [ (log pi(chosen) - log ref(chosen)) - (log pi(rejected) - log ref(rejected)) ] )

beta sets how hard the tuned model is held near the reference. Two measures afterwards: the share of word gaps written as underscores on 60 new prompts (the style), and the loss on held-out English counting only letters, not gaps (the damage).

Words and symbols in this lesson
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
rank (r)
how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
DPO
direct preference optimisation: tuning from pairs of a better and a worse answer
reference model
the frozen model DPO measures every change against
beta (DPO)
how hard the tuned model is held near the reference
Your turn

Choose steps (at most 60) so the style is learned (underscore share at least 0.8) while the letters-only loss stays under 2.6 (the base model scores about 1.94). Try a few values.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.