Prefill and Decode

Prefill and decode are the two distinct computational phases of LLM inference. Prefill is the phase in which the model processes the full input prompt to build its initial internal state. Decode is the phase in which the model generates output tokens one at a time, each step conditioned on the accumulated state from the prompt and all previously generated tokens. The two phases have fundamentally different computational profiles and memory access patterns, and inference systems treat them differently at the scheduling, kernel, and hardware levels.

How do prefill and decode work?

During prefill, the model ingests all prompt tokens in parallel. For transformer-based models, this means computing key-value (KV) pairs for every prompt token and storing them in the KV cache. For SSM-based architectures like Mamba, prefill computes an initial compressed hidden state that summarizes the entire prompt. Because prefill processes a sequence for the first time, the hidden state must be initialized to zero before computation begins. During decode, the model processes one new token per step. Transformer decode reads previously cached KV pairs and attends over them; Mamba decode loads the hidden state from its cache slot and updates it recurrently with the new token before writing it back. The key operational difference is directionality: transformer decode is a write-then-read operation (the new token’s KV is written to cache before attention reads over all cached tokens), whereas Mamba decode is a read-then-update operation (existing state is loaded before the new token modifies it). This asymmetry means that misclassifying a new prefill request as a decode is catastrophic for Mamba – the model reads stale state from a prior request’s cache slot before the new request has had any opportunity to initialize – while the same misclassification in a transformer is self-correcting, because the new token’s KV is written to cache before the read occurs.

What are prefill and decode used for?

Understanding the distinction between prefill and decode is essential for designing and operating efficient LLM inference systems. Inference schedulers such as vLLM use the phase classification to route requests to the correct computational kernels, manage cache allocation, and optimize batch composition. Decode-heavy workloads – such as long-form generation and multi-turn dialogue – benefit from continuous batching, which groups many single-token decode steps together to maximize GPU utilization. Prefill-heavy workloads – such as document summarization or retrieval-augmented generation with long contexts – benefit from chunked prefill and prefix caching strategies. The separation also underlies advanced serving architectures like prefill-decode disaggregation, where dedicated hardware handles each phase independently to improve overall system throughput. For teams building or debugging inference pipelines for hybrid architectures such as Jamba, correctly classifying every request as prefill or decode at every scheduling step is a correctness requirement, not merely a performance optimization.