Generating fast: the key-value cache
Why it matters Chatbots write one token at a time. Done naively, every new token recomputes the keys and values of the whole text so far. The key-value cache is why replies stream quickly.
See it
Play the animation. With the causal mask, the keys and values of earlier positions never change when new tokens arrive. So keep them: for each new token compute only its own q, k and v, append k and v to the cache, and attend with q over the cached keys.
Work per new token: the naive way computes t keys and values for a text of length t; the cache computes 1. Over a 1,000-token reply that is about 500,000 key computations against 1,000.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
- self-attention
- every position attends to the positions of the same text
- causal mask
- scores above the diagonal set to minus infinity: no position sees the future
- key-value cache
- keep earlier keys and values so each new token costs one step
Write cached_step(x, cache, Wq, Wk, Wv): x is one new row; append its key and value to cache["K"] and cache["V"] (lists), and return its attention output over all cached positions. Feeding the rows of X one at a time must reproduce causal self-attention exactly. Set keys_naive and keys_cached: how many key vectors each method computes to produce 1,000 tokens.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.