KV cache (Key-Value cache) stores the Key and Value vectors the model computed for previous tokens, so that generating a new token only requires computing Q, K, V for that new token and reusing the cached K/V from everything before it — instead of recomputing attention over the entire context from scratch at every single generation step.

Without KV cache, generating token N would mean recomputing attention over tokens 1 through N-1 all over again — step 2 recomputes token 1+2, step 3 recomputes token 1+2+3, and so on, which is wasteful and gets slower as the conversation grows. With KV cache, only the new token's Q/K/V need computing, and everything else is reused from the cache, which improves inference speed significantly and is especially important for long context, chat applications, streaming generation, low-latency systems, and production serving generally. The trade-off is memory: KV cache consumes GPU memory proportional to context length and batch size, so longer conversations and larger batches require substantially more memory to keep the cache resident.