LoRA, built
Why it matters Building it once makes three things obvious: why it starts out harmless, why its gradients are cheap, and why the original weights never change.
A LoRA layer on the house engine: forward is x W + (alpha / r) x A B, with W frozen. The engine works out the gradients for A and B by itself (your Module 4 engine would too); W just must not be updated, which you check.
Words and symbols in this lesson
- LoRA
- low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
Write lora_forward(x, W, A, B, alpha) with engine operations. Then check, on the small layer in the starter: the output equals x W while B is zero; after one backward pass, W has no update but A or B does; and the gradient of A matches nudging. Set max_gap to the largest difference. The engine works in float32, so nudge by h = 1e-3 or more: with 1e-4 the change in the loss is lost in rounding and a correct gradient looks wrong.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.