Continuous Batching

What is Continuous Batching?

Continuous batching is a scheduling technique for large language model inference that allows new requests to join an active generation batch as soon as a slot becomes available, rather than waiting for all requests in the current batch to finish before starting the next one. It is a key mechanism enabling high GPU utilization in production LLM serving systems, and is the default batching strategy in frameworks such as vLLM. Continuous batching is sometimes called iteration-level scheduling because the decision about which requests to include is made at each generation step rather than once per batch.

How does Continuous Batching work?

In traditional static batching, the server collects a fixed set of requests, runs them through the model together until all have finished generating, and only then accepts new requests. This creates a padding problem: requests that finish early sit idle while the longest request in the batch completes, wasting compute. Continuous batching eliminates this waste by treating each forward pass through the model as an opportunity to update the batch composition. At every decode step, the scheduler checks the request queue and inserts waiting requests into any slots freed by completed sequences, subject to memory and batch-size limits set by parameters such as –max-num-seqs (maximum concurrent sequences) and –max-num-batched-tokens (maximum tokens processed per forward pass). The result is a nearly full GPU at all times, with new requests interleaved into ongoing generation work rather than queued behind it.

What is Continuous Batching used for?

Continuous batching is the primary mechanism for achieving high throughput in online LLM inference serving, particularly under variable or bursty request patterns. For workloads where request arrival is unpredictable, such as multiple concurrent training jobs competing for a shared LLM-as-a-Judge deployment, continuous batching allows the serving system to absorb large spikes of incoming requests into a single active batch rather than queuing them all behind a fixed batch boundary. Configuring it correctly requires matching the –max-num-seqs and –max-num-batched-tokens parameters to the actual workload characteristics, including typical sequence lengths and traffic burstiness. Misconfigured values, such as the conservative defaults of 64 sequences and 2,048 batched tokens, can leave most GPU capacity unused even when the request queue is long. Properly tuned continuous batching is often the single largest contributor to throughput improvements in vLLM deployments.