Your engine, in PyTorch terms
Why it matters You will meet PyTorch (or another framework) the day after this course. Everything you built maps onto it; only a few conventions differ, and they cause most of the bugs when people port code.
PyTorch cannot run in this browser, so this lesson converts the course's arrays into PyTorch's layouts with NumPy, and checks the conversion exactly.
| Thing | This course | PyTorch |
|---|---|---|
| Linear weight | W is (n_in, n_out), output x @ W + b | weight is (out, in), output x @ weight.T + bias |
| Starting weights | normal with spread sqrt(2 / n_in) (He) for every layer, bias 0 | nn.Linear default: uniform within about ±1 / sqrt(n_in), bias uniform too |
| Images | channels last, (N, H, W, C) | channels first, (N, C, H, W) |
| Conv weight | (k × k × C_in, C_out), rows ordered by (row, column, channel) | (C_out, C_in, k, k) |
| BatchNorm running variance | the batch variance dividing by n | the unbiased one, dividing by n - 1 (normalising still uses n) |
| Clearing gradients | the house Adam.step() clears them itself | you call optimizer.zero_grad() every step |
Which tensors keep .grad | every node of the graph | only leaf tensors such as weights, unless you call retain_grad() |
| GELU | z * sigmoid(1.702 z), an approximation | the exact form with erf by default; approximate='tanh' for the fast one |
| LSTM | one weight matrix on [x, h], forget bias started at 1 or 3 | separate input and hidden weights with two biases, all started small and random |
| Tiny GPT block | ReLU, no biases in its linear layers: 12d² + 4d weights | GPT-2: GELU and biases: 12d² + 13d |
None of these is wrong; they are choices. The ones that silently break a port are the layouts: a transposed Linear weight or a channels-last image gives wrong numbers with no error when the sizes happen to match.
Words and symbols in this lesson
- head
- one of H narrower attentions run side by side, each with its own pattern
Write linear_to_torch(W, b) returning PyTorch's (weight, bias), nhwc_to_nchw(X), conv_to_torch(w, c_in) returning the (C_out, C_in, k, k) weight, and unbiased(var, n) turning a variance that divided by n into one that divides by n - 1. The hidden test runs a channels-first convolution written the PyTorch way with your weights and compares it with the course's conv.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.