Online Reinforcement Learning (Online-RL) Training

Online reinforcement learning (online-RL) training is a method for training large language models in which the model actively generates its own responses during training and receives feedback – in the form of reward signals – that is used to update its parameters in near real time. Unlike supervised fine-tuning, which trains on a fixed dataset of labeled examples, online-RL creates a continuous loop: the model produces outputs (called rollouts), a reward model or automated judge evaluates those outputs, and the model’s policy is updated based on the resulting reward. This iterative process allows the model to improve its reasoning, instruction-following, and task-completion abilities through direct trial-and-error feedback rather than imitation of pre-labeled data.

How does online-RL training work?

A standard online-RL training loop consists of three repeating phases. In the rollout generation phase, the model receives a batch of prompts and generates responses using its current policy weights. These responses are typically sampled stochastically to encourage exploration of different output strategies.

In the reward computation phase, the generated responses are scored by a reward model – often a separate large language model trained to evaluate quality – or by a verifiable criterion such as mathematical correctness. The reward signal captures how well each response satisfies the desired objective, whether that is factual accuracy, reasoning quality, or instruction adherence.

In the policy update phase, the reward signals are used to compute gradients and update the model’s weights so that higher-reward responses become more probable in future iterations. Algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) are commonly used at this stage to ensure that weight updates remain stable and do not cause the policy to drift too far from its previous behavior in a single step.

Because the model both generates data and trains on it within each cycle, online-RL runs are long-lived and compute-intensive relative to static fine-tuning. Inefficiencies in any phase – including padding waste in the policy update step – compound across thousands of iterations and can meaningfully increase total training cost and wall-clock time.

What is online-RL training used for?

Online-RL training is used to develop models with strong reasoning and instruction-following capabilities that are difficult to instill through supervised learning alone. It is the primary training method behind models optimized for mathematical problem solving, code generation, multi-step reasoning, and complex question answering.

Frameworks such as VeRL (Versatile Reinforcement Learning) have been developed to support online-RL training of large language models at scale, providing infrastructure for distributed rollout generation, reward computation, and policy updates across multiple GPU nodes. These frameworks are typically designed around transformer architectures, which means that non-transformer and hybrid models may encounter efficiency gaps – such as padding waste in the policy update step – that require model-agnostic infrastructure solutions to address.