GLOSSARY
KV Cache
The memory that lets a model keep notes about tokens it has already read instead of recomputing them — the reason long conversations eat GPU memory.
Transformers attend over everything seen so far. Recomputing all of that for every new token would be ruinously slow, so serving engines store each token's key/value projections — the KV cache — and reuse them. The cache grows with conversation length and batch size, which is why serving capacity is measured in memory as much as compute, and why context windows cost money.
The famous innovation here is vLLM's PagedAttention: managing the KV cache like an operating system manages RAM, in pages instead of contiguous blocks — cutting waste dramatically and raising throughput for everyone downstream. When an API charges differently for long contexts, cache memory is the physics underneath the pricing page.