KV cache

The model’s scratch notes from earlier in the conversation, kept so it doesn’t have to re-read everything from the start for every new word. It makes answers fast and eats a lot of memory.

A gouache picture-book scene: a boy reads in a yellow armchair beside a teetering tower of sticky notes that climbs all the way up to the lamp.

When an LLM writes, it produces one token at a time, and each new token needs to look back at every token before it through attention . Done naively, that means re-processing the entire conversation for every single word. The KV cache avoids this: as each token is processed, the model saves two small sets of numbers for it, the keys and values that attention uses, and reuses them for every token that follows.

That makes inference dramatically faster, and it’s why the first word of a reply can take a moment while the rest stream quickly: the model is filling the cache with your prompt first.

The cost is memory. The cache grows with every token in the context window , so a long conversation can need gigabytes for the cache alone, on top of the model itself. On a laptop, it’s often the cache, not the model, that decides how long a conversation can get. Local runners can quantize the cache to shrink it.