Teaching preferences (DPO)
Why it matters Chat models are not only trained on text; they are tuned on preferences: of two answers, which one people liked better. DPO (Rafailov et al. 2023) does it directly from pairs, with one loss and no separate reward model. It is also easy to overdo.
The preference here is a style the base model never uses: joining words with underscores ("snake_case talk"). Each pair has a 20-character prompt from the lesson text, the real next 10 characters (rejected) and the same 10 with every space written as an underscore (chosen).
The DPO loss for one pair compares how much more likely the tuned model makes the chosen answer than the reference model did, against the same for the rejected one:
-log sigmoid( beta x [ (log pi(chosen) - log ref(chosen)) - (log pi(rejected) - log ref(rejected)) ] )
beta sets how hard the tuned model is held near the reference. Two measures afterwards: the share of word gaps written as underscores on 60 new prompts (the style), and the loss on held-out English counting only letters, not gaps (the damage).
Words and symbols in this lesson
- LoRA
- low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
- rank (r)
- how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
- DPO
- direct preference optimisation: tuning from pairs of a better and a worse answer
- reference model
- the frozen model DPO measures every change against
- beta (DPO)
- how hard the tuned model is held near the reference
Choose steps (at most 60) so the style is learned (underscore share at least 0.8) while the letters-only loss stays under 2.6 (the base model scores about 1.94). Try a few values.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.