KV Cache

What is a KV Cache?

A KV cache (key-value cache) is a memory structure used in large language model inference to store the intermediate attention computations generated while processing a sequence of tokens. Rather than recomputing these values from scratch for each new token, the model reads them directly from cache, dramatically reducing the computational work required to generate each successive output token. The size of the KV cache is one of the primary constraints on how many requests an LLM serving system can handle concurrently.

How does a KV Cache work?

During the prefill phase of LLM inference, the model processes all input tokens in parallel and computes key and value tensors for each attention head at every transformer layer. These tensors are written into GPU memory as the KV cache. During the decode phase, when the model generates output tokens one at a time, it reads from this cache rather than reprocessing the full input sequence. Each newly generated token is appended to the cache so subsequent tokens can attend to the growing context without repeated computation. Because the cache lives in GPU VRAM, its size is bounded by available GPU memory. Inference frameworks like vLLM pre-allocate a fixed block of VRAM for the KV cache at startup, controlled by parameters such as gpu-memory-utilization, and use paged memory management to allocate cache slots to requests dynamically as they arrive.

What is a KV Cache used for?

KV caching is a foundational optimization in any production LLM serving system. Without it, generating a 500-token response from a 2,000-token prompt would require running the full attention computation over all 2,000 input tokens for each of the 500 output steps. With caching, that computation happens once. The amount of VRAM allocated to the KV cache directly determines how many concurrent sequences a single GPU or GPU group can hold, making it a central factor in throughput capacity planning. In multi-GPU deployments using tensor parallelism, the KV cache is distributed across GPUs, which is one reason pooling more GPUs per instance can unlock significantly more concurrent request capacity. Managing KV cache pressure, through techniques like prefix caching, cache eviction policies, and paged attention, is a primary concern in high-throughput LLM inference engineering.