Sequence Packing

Sequence packing is a training optimization technique in which multiple variable-length input sequences are concatenated into a single longer sequence before being passed to a model, eliminating the need for padding tokens. Instead of padding each sequence to a fixed maximum length to fill a rectangular batch tensor, packing fills the available token budget by chaining sequences end-to-end. This removes the wasted compute that arises when a model processes dummy padding tokens that carry no meaningful information, and is one of the most effective methods for improving training throughput on datasets with heterogeneous sequence lengths.

How does sequence packing work?

In standard batched training, sequences of different lengths are padded to match the longest sequence in the batch, producing a dense rectangular tensor in which many positions contain dummy tokens. Sequence packing instead treats the available sequence length as a token budget and fills it by appending sequences one after another until the budget is exhausted, leaving little or no empty space.

The key technical challenge is ensuring that tokens from one sequence do not influence tokens from another during the attention computation. Token-wise operations such as MLP layers and layer normalization are position-independent and naturally treat each token in isolation, so packing does not affect them. Attention layers, however, allow tokens to exchange information across positions. Without special handling, a token at the start of a packed second sequence could attend to tokens from the first, introducing cross-sequence contamination that corrupts training.

Modern efficient attention implementations such as FlashAttention support document masking, which enforces sequence boundaries during attention computation. Rather than applying a naive mask – which still incurs quadratic cost – these implementations restrict attention to tokens within the same original sequence, achieving the efficiency of packing without any cross-sequence leakage. This makes sequence packing safe for use in online-RL training, where ensuring that the model sees consistent conditioning at generation time and training time is critical to stable learning.

Importantly, sequence packing is an architecture-specific optimization. It requires each model type to implement its own correct handling of packed sequence boundaries. For transformer models, mature implementations exist. For state-space and hybrid architectures such as Mamba, adding packing support requires non-trivial engineering, since the recurrent state update mechanism does not natively distinguish between sequences that have been concatenated along the sequence dimension.

What is sequence packing used for?

Sequence packing is used to improve training throughput and GPU utilization in large language model pre-training, supervised fine-tuning, and reinforcement learning pipelines. It is especially valuable on datasets with highly variable sequence lengths – such as instruction-following or reasoning datasets – where standard padding would leave a large fraction of processed tokens carrying no useful information.

In practice, sequence packing can reduce policy update step times by as much as 70-80% relative to default padded batching on transformer models, making it one of the highest-impact single optimizations available in LLM training infrastructure. Its absence in hybrid and non-transformer architectures – where implementing it correctly requires substantial additional engineering – has motivated the development of model-agnostic alternatives such as micro-batch-level padding truncation and padding-aware dynamic micro-batching, which can recover most of the efficiency gain without requiring architecture-level changes.