TL;DR

When running GRPO on verifiable rewards, dynamic sampling does a lot to stabilize the training process. However, it may also slow training down dramatically. In this blog, we introduce “dynamic data snoozing” as a simple and effective strategy for cutting compute waste without breaking training stability. By applying this method, we are able to achieve up to a 3X gain in compute efficiency with no quality degradation.

Efficient batch construction for GRPO

GRPO, the de-facto RL algorithm used today due to its effectiveness and simplicity, often runs into cases where a training input results in no learning signal. This can happen when all rollouts for that input have the same reward, so the computed advantage of each rollout is 0. Ignoring this fact when running GRPO leads to training instabilities, as our effective batch size isn’t fixed and depends on the amount of examples with a learning signal. 

These 0-advantage examples cause training instabilities, and this problem was addressed in the DAPO paper with the now common-practice mechanism of dynamic sampling. With dynamic sampling, groups without a learning signal which are either too-easy or too-hard are filtered and replaced with a new input. Regardless of whether 0-advantage examples are practically ignored by the training process or explicitly replaced, they significantly slow down the training process. We create wasteful generations for them which have no impact on the model’s training.

Figure 1 shows the total number of examples used for generation during the DAPO training run, vs. the number of examples actually used for gradient computation. The training inefficiency is apparent:

  • Initially: Training efficiency is reasonable since many examples are in the range of not-too-easy and not-too-hard. 
  • As training progresses: Training slows considerably by up to 16X compared to an optimal example sampler, as examples become too easy.

At the end of the day, less than 15% of the generations performed in the training run are actually used for training.

When Sleeping In Saves You Money: Dynamic Data Snoozing for Efficient Online RL
Figure 1: Training efficiency of DAPO, using dynamic sampling. We can see that as training progresses, the gap between the number of generated batches vs. the optimal amount of generated batches (one per step) increases, leading to slower and less efficient training.

To optimize this large gap in performance, we developed a few simple methods that gave us performance gains during the training of Jamba2.

Offline difficulty stratification

We began with offline difficulty stratification, a critical preparatory step in online RL training, as it provides a grounded understanding of task complexity and model behavior before learning begins. This mapping can inform both data filtering and curriculum building.

To create this stratification, we ran multiple completions of a few baseline models on our training data. Based on the success rate of these models on each example, we were able to partition our training data based on empirical difficulty and build a curriculum for our model: Overly-difficult examples, that would have been filtered anyway, were postponed to a later stage of training. On the other hand, overly-easy examples could be removed entirely from the training process when the time was right. Using this stratification, we could also adapt the dataset weights to upsample sections of the data with the most signal for the current model checkpoint.

While useful for manually controlling the difficulty of examples seen by the model during training, this off-policy method leads to a suboptimal categorization of example difficulty. To more effectively handle the overly-easy side of the data, we turned to an online, on-policy method.

Dynamic difficulty filtering

To optimize the removal of overly-easy examples from our training sets and to simplify the process of data rebalancing, we introduced a simple but effective heuristic to our training procedure.

Assume the reward for a given dataset is in the range of [0, 1]. An example is defined as too-easy if all rollouts in its group have a reward of 1. An example like this would be filtered out since all rollouts will have an advantage of 0. Assuming training goes well, our model should continue to get a perfect score on this example, so there’s no reason to keep showing it to our model in future batches. So, once an example gets filtered out for being too easy, we keep filtering it out every time it reappears in subsequent epochs, saving the expensive rollout generation phase. The training budget can then be reallocated for appropriately-difficult examples.

Figure 2 describes the training dynamics when we apply Dynamic difficulty filtering to the GSM8K dataset. Dynamic filtering leads to a considerable 3X gain in compute efficiency. The final test scores remain similar, with all variants reaching a score of 84%.

When Sleeping In Saves You Money: Dynamic Data Snoozing for Efficient Online RL
Figure 2: Training efficiency of GSM8K. We compare DAPO dynamic sampling (red) with our proposed methods – dynamic difficulty filtering (blue) and dynamic data snoozing with a snoozing factor of 5 (green, introduced later in the post). We see both methods greatly reduce the number of batches generated during training compared to naive dynamic sampling.

Maintaining efficiency in multi-task batch construction

While dynamic difficulty filtering provides considerable efficiency gains, it is not without its flaws. On a simple dataset like GSM8K there are no tradeoffs, but things become messier when we deal with the realistic scenario of training on multiple datasets1. Since different tasks are learned at different rates, the rate of filtering from each dataset is different and changes over time. This makes the balance of datasets within a batch harder to control, and this can be detrimental to stability when using online difficulty filtering. We needed to find a way to maintain our efficiency gains, while maintaining stability across multiple datasets. 

Task starvation and mode collapse: Risks of dynamic difficulty filtering

In Figure 3 we can see a math dataset stratified by empirical difficulty into easy and hard subsets. The two datasets are equally-weighted when sampling before batch construction. The figure describes the fraction of each dataset in the batch in each step. First, due to the different group-filtering rates, we observe how they are not equally represented in the batch. As training progresses, the batch gradually becomes even more dominated by the harder subset, as the model becomes better at solving math tasks.

When Sleeping In Saves You Money: Dynamic Data Snoozing for Efficient Online RL
Figure 3: Training batch proportions during training with DAPO dynamic sampling. As training progresses and the model becomes more capable, the easy subset of the training set is filtered more often as it becomes too easy.

For large multi-task settings, the above may cause unexpected problems during training. One potential problem is starvation – if a task is relatively harder compared to other tasks, it could be filtered more than other datasets and become under-represented in training. Datasets like these will improve more slowly than others unless we reserve some minimum amount of examples for them in each batch. 

Another problem arises for datasets with conflicting tasks that require careful balancing. Some examples include:

  1. Safety-related tasks, where we want to teach the model to refuse the user request when it is inappropriate in some way. 
  2. Grounded question-answering, where we want the model to only answer based on the provided context, and refrain from answering if the answer doesn’t appear in the context.
  3. Open-ended question-answering, where we want the model to answer based on its knowledge cutoff or otherwise use a web search tool.

For these tasks, if the balance of inputs in the batch becomes too skewed for too long, the model may collapse to a one-sided behavior like constantly refusing to answer. 

To demonstrate the challenges of conflicting tasks, we train our model on a grounded question-answering dataset for which some of the inputs cannot be answered. We set the weights of the two datasets as 10% for non-answerable questions and 90% for answerable ones. In Figure 4.C we see the actual ratio of non-answerable questions in the batch as training progresses, comparing regular training and online difficulty filtering. Due to the training dynamics, many answerable questions get filtered and we are left with an over-abundance of non-answerable questions in our training set and batches. Figures 4.A and 4.B show the performance on each subset of the data. The increased representation of non-answerable questions in the batches leads to a clear degradation in performance on the answerable subset, as the model begins to refuse to answer many of the answerable questions.

To respond to these potential risks, we needed a method that walks a middle ground: removing overly-easy examples so training doesn’t slow down, while ensuring the training data remains balanced even across multiple datasets.

Introducing dynamic data snoozing

Reviewing the data above on conflicting tasks, it’s clear that the assumption that an easy example remains easy forever is invalid. We need a way to check-in on these examples to make sure they aren’t forgotten. To support this capability, while still maintaining the performance gains the online filtering gives us, we adapted dynamic difficulty filtering into dynamic data snoozing. 

Simply put, instead of permanently filtering an overly-easy example, we “snooze” it for a predefined number of epochs. Figure 4 shows the performance of the same dataset with a snoozing factor of 2, 3 and 5 epochs. We see that higher snoozing factors lead to higher temporary degradations in performance during training, yet the model is able to recover. This demonstrates the viability of our dynamic data snoozing method to maintain the performance gains of dynamic filtering, while avoiding the potential pitfalls of the multi-task settings.

Figure 4: Comparison of example sampling approaches on a grounded question-answering dataset. Dynamic snoozing maintains strong performance on both answerable and non-answerable subsets (A and B, respectively) while remaining computationally efficient. In contrast, aggressive filtering (in red) leads to training imbalance and mode collapse. The snoozing factor governs the tradeoff between computational efficiency and risk of training instability.

Figure 2 shows the performance of dynamic snoozing on the GSM8K dataset. While the more aggressive dynamic filtering gives an efficiency gain of 3X, dynamic snoozing still gives a solid 2X gain.

Dynamic data snoozing in practice

Choosing the snoozing factor

The snoozing factor represents a tradeoff of how much compute we want to save on a given dataset vs. the risk of training imbalance due to the dataset’s interaction with other tasks. If we have a solid understanding of the dataset we’re training on and believe it is relatively standalone, we choose a higher snoozing factor to save more compute. Alternatively, if we believe there are many interactions within the dataset, we choose a snoozing factor of 1 or 2. By default, we use a snoozing factor of 5 and see overall good results with big compute gains.

Fixing batch proportions

In this blogpost we assume dataset weights are defined before filtering, such that if an example is filtered we sample a new example with these same weights. Alternatively, we could have defined the weights over the final batch, resampling filtered examples from the same original dataset.

Making this adaptation has some efficiency costs, but improves many of the issues above. However, note that often we cannot assume that there is a clear split between two conflicting datasets. A single dataset can be implicitly composed of different distributions, and snoozing allows us to prevent the actual batch distribution from skewing.

Smoothing out the instabilities with probabilistic snoozing

Looking back at figure 4, we see that snoozing allows training to progress, but there are still instabilities and drops for large snoozing factors. To smooth this behavior out and make training more consistent and stable, we experimented with a probabilistic version of snoozing. Instead of snoozing an example of N epochs, we filter it at each epoch with a probability of 1/N. This approach generally works and is a reasonable alternative to deterministic snoozing.

Dynamic example weighting

The probabilistic snoozing above can be seen as simply decreasing the weight of examples that are too easy. Given this new framing, one can extend the dynamic weighting to all training examples, not just the ones with no learning signal. The empirical signal we get from our model’s performance for each example can be taken into account when assigning sampling weights to individual examples, increasing the weight of examples that are most appropriate for our model. We’ve been experimenting with this direction and see a lot of potential for it.

Conclusion

We have explored how careful treatment of data difficulty and curriculum construction can greatly improve the efficiency of current online RL training methods. As we move to training LLMs on agentic, long-horizon tasks, online RL is gradually becoming the most expensive part of the entire LLM training stack. We identify data efficiency as a largely overlooked avenue of exploration, with the potential to considerably increase the training efficiency of our pipeline.

We showcase this with dynamic data snoozing, a simple and straightforward online adaptation to our data sampling that achieves significant efficiency improvements. We also discuss the nontrivial dynamics of online RL training in realistic multi-task settings, where overly-naive methods may result in large training instabilities in certain cases.

We hope this work will spark further interest in this research direction.

Citation

@misc{ai21labs2026snoozing,

  title={{When Sleeping In Saves You Money: Dynamic Data Snoozing for Efficient Online RL}},

  author={Daniel Gissin, Yuval Globerson and Inbal Magar},

  year={2026},

  howpublished={\url{https://www.ai21.com/blog/dynamic-data-snoozing/}}

}

Footnote

  1.  Note that we assume multiple tasks are learned at the same time within the same batches. There are alternative multi-task learning settings like sequential/continual learning, or learning in parallel and merging model weights. While these methods have merit, we’ve found learning tasks together to be necessary for optimal performance.
     ↩︎