GRPO (Group Relative Policy Optimization)

GRPO, or Group Relative Policy Optimization, is a reinforcement learning algorithm used to fine-tune large language models (LLMs) based on outcome-based reward signals. It is designed to be simpler and more computationally efficient than earlier RL methods such as PPO (Proximal Policy Optimization), while achieving comparable or superior results on reasoning and instruction-following tasks. GRPO has become one of the most widely adopted algorithms in modern LLM post-training pipelines.

How it works

In GRPO, the model generates a group of multiple completions (rollouts) for each training input. Each completion is scored using a reward function – for example, checking whether the answer to a math problem is correct. The algorithm computes the relative advantage of each completion within its group by comparing individual rewards to the group mean. This group-relative advantage is used to update the model’s policy: completions that scored above the group average are reinforced, while those that scored below are discouraged. Because GRPO estimates the baseline from the group itself rather than requiring a separate value network (as PPO does), it eliminates a significant source of computational overhead.

A known challenge in GRPO arises when all completions in a group receive the same reward – typically all correct or all incorrect. In these cases, the computed advantage for every rollout is zero, meaning the batch contributes no gradient signal to the model. Filtering or rescheduling such zero-advantage groups is a key motivation for techniques like dynamic data snoozing.

What it is used for

GRPO is used in the post-training phase of LLM development, particularly for tasks with verifiable outcomes such as mathematical reasoning, coding, and structured question answering. It enables models to improve through trial and error by generating candidate answers and learning from which ones are rewarded. GRPO is the core algorithm behind several high-profile open models trained with reinforcement learning from verifiable rewards, and it forms the foundation of many modern alignment and reasoning pipelines.