Syllabus Lesson 157 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Your engine, in PyTorch terms

Why it matters You will meet PyTorch (or another framework) the day after this course. Everything you built maps onto it; only a few conventions differ, and they cause most of the bugs when people port code.

PyTorch cannot run in this browser, so this lesson converts the course's arrays into PyTorch's layouts with NumPy, and checks the conversion exactly.

ThingThis coursePyTorch
Linear weightW is (n_in, n_out), output x @ W + bweight is (out, in), output x @ weight.T + bias
Starting weightsnormal with spread sqrt(2 / n_in) (He) for every layer, bias 0nn.Linear default: uniform within about ±1 / sqrt(n_in), bias uniform too
Imageschannels last, (N, H, W, C)channels first, (N, C, H, W)
Conv weight(k × k × C_in, C_out), rows ordered by (row, column, channel)(C_out, C_in, k, k)
BatchNorm running variancethe batch variance dividing by nthe unbiased one, dividing by n - 1 (normalising still uses n)
Clearing gradientsthe house Adam.step() clears them itselfyou call optimizer.zero_grad() every step
Which tensors keep .gradevery node of the graphonly leaf tensors such as weights, unless you call retain_grad()
GELUz * sigmoid(1.702 z), an approximationthe exact form with erf by default; approximate='tanh' for the fast one
LSTMone weight matrix on [x, h], forget bias started at 1 or 3separate input and hidden weights with two biases, all started small and random
Tiny GPT blockReLU, no biases in its linear layers: 12d² + 4d weightsGPT-2: GELU and biases: 12d² + 13d

None of these is wrong; they are choices. The ones that silently break a port are the layouts: a transposed Linear weight or a channels-last image gives wrong numbers with no error when the sizes happen to match.

Words and symbols in this lesson
head
one of H narrower attentions run side by side, each with its own pattern
Your turn

Write linear_to_torch(W, b) returning PyTorch's (weight, bias), nhwc_to_nchw(X), conv_to_torch(w, c_in) returning the (C_out, C_in, k, k) weight, and unbiased(var, n) turning a variance that divided by n into one that divides by n - 1. The hidden test runs a channels-first convolution written the PyTorch way with your weights and compares it with the course's conv.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.