Syllabus Lesson 210 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

LoRA, built

Why it matters Building it once makes three things obvious: why it starts out harmless, why its gradients are cheap, and why the original weights never change.

A LoRA layer on the house engine: forward is x W + (alpha / r) x A B, with W frozen. The engine works out the gradients for A and B by itself (your Module 4 engine would too); W just must not be updated, which you check.

Words and symbols in this lesson
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
Your turn

Write lora_forward(x, W, A, B, alpha) with engine operations. Then check, on the small layer in the starter: the output equals x W while B is zero; after one backward pass, W has no update but A or B does; and the gradient of A matches nudging. Set max_gap to the largest difference. The engine works in float32, so nudge by h = 1e-3 or more: with 1e-4 the change in the loss is lost in rounding and a correct gradient looks wrong.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.