Syllabus Lesson 156 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Generating fast: the key-value cache

Why it matters Chatbots write one token at a time. Done naively, every new token recomputes the keys and values of the whole text so far. The key-value cache is why replies stream quickly.

See it

Play the animation. With the causal mask, the keys and values of earlier positions never change when new tokens arrive. So keep them: for each new token compute only its own q, k and v, append k and v to the cache, and attend with q over the cached keys.

Work per new token: the naive way computes t keys and values for a text of length t; the cache computes 1. Over a 1,000-token reply that is about 500,000 key computations against 1,000.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
self-attention
every position attends to the positions of the same text
causal mask
scores above the diagonal set to minus infinity: no position sees the future
key-value cache
keep earlier keys and values so each new token costs one step
Your turn

Write cached_step(x, cache, Wq, Wk, Wv): x is one new row; append its key and value to cache["K"] and cache["V"] (lists), and return its attention output over all cached positions. Feeding the rows of X one at a time must reproduce causal self-attention exactly. Set keys_naive and keys_cached: how many key vectors each method computes to produce 1,000 tokens.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.