Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.
Figures & tables
Figure 1: Overview of our work. (Top) Reward saturation in GRPO: When training data are easy, all G rollouts could be correct and receive the maximum reward, leading to zero advantage and thus little policy update. (Bottom) Multi-Level Interventions: We intervene at one of four stages of GRPO training: data, rollout, reward, and advantage to restore informative learning signals from reward-saturated examples. These interventions act at different stages of the training pipeline but share the same goal: restoring a usable group-relative learning signal for otherwise saturated prompts.
Settings
MATH500
GPQA
BBH
Minerva
AIME24
AIME25
IFBench
IFEval
Average
Policy: Qwen3-1.7B
Base Model
60.00
25.80
9.65
18.00
11.46
9.06
23.20
68.00
28.15
Baselines
GRPO
62.60
24.24
10.86
17.28
17.08
9.17
21.20
69.00
28.93
Mixed-CUTS
61.00
32.32
10.55
18.38
16.77
12.81
25.20
70.20
30.91
SFT
58.40
29.29
8.23
18.01
6.67
10.10
22.00
67.40
27.51
Table 1: Performance comparison across eight benchmarks using Qwen3-1.7B and Qwen3-4B. We compare the base model, standard GRPO, Mixed-CUTS, SFT, SFT → GRPO and interventions at the data, rollout, reward, and advantage levels. We report avg@32 for AIME24 and AIME25 and pass@1 for all other benchmarks. The best and second-best results within each model group are highlighted in bold and underline , respectively.
Method
MATH500
GPQA
BBH
Minerva
AIME24
AIME25
IFBench
IFEval
Average
Qwen3-1.7B
GRPO
62.60
24.24
10.86
17.28
17.08
9.17
21.20
69.00
28.93
Answer Replacement
59.20
27.78
13.10
14.71
2.50
7.29
29.20
67.80
27.70
Wrong-Answer-Cond.
64.20
29.80
10.86
17.28
17.19
9.17
23.20
70.60
30.29
Model-Generated Negative
64.80
29.80
13.01
18.38
20.31
14.38
24.00
67.60
31.53
Qwen3-4B
Table 2: Comparison of negative rollout construction strategies across eight benchmarks. Although all three strategies restore reward variation within saturated groups, model-generated negative rollouts achieve the strongest average performance across both model scales.
Figure 2: Performance of GRPO and Negative Rollout across different ratios of reward-saturated training data. Negative Rollout improves aggregate performance across all saturation ratios, with gains varying across benchmarks.
Figure 3: Self-evolving saturation-aware training, with expanding saturated pool and improving AIME25 performance.
Settings
MATH500
GPQA
BBH
Minerva
AIME24
AIME25
IFBench
IFEval
Average
GRPO
70.60
31.31
16.13
25.74
31.98
27.40
28.00
78.80
38.74
Zero Padding ( k=1 )
68.20
34.85
15.11
23.90
21.67
19.27
29.20
80.60
36.60
Zero Padding ( k=2 )
62.80
31.82
13.83
19.85
14.06
11.88
27.60
81.40
32.90
Zero Padding ( k=3 )
67.20
30.30
13.50
22.79
20.21
23.13
30.00
79.80
35.87
Zero Padding ( k=8 )
62.20
29.80
13.50
22.79
17.71
3.65
29.60
79.60
32.36
Zero Padding (random k )
66.40
30.30
13.27
22.43
17.08
18.96
27.60
80.00
34.51
Table 3: Effect of number of zero-reward padding size k on Qwen3-4B.
Settings
MATH500
GPQA
BBH
Minerva
AIME24
AIME25
IFBench
IFEval
Average
Base Model
34.00
18.18
8.96
12.50
6.25
1.15
30.80
72.00
22.98
GRPO
45.00
20.71
10.42
13.97
3.30
1.04
32.90
72.40
24.97
Mixed-CUTS
27.00
3.03
2.50
9.93
3.30
0.21
29.20
63.60
17.35
Negative Rollout
43.60
24.75
9.27
20.59
10.00
0.42
29.20
67.20
25.63
Table 4: Generalization results on LLaMA-3.1-8B-Instruct across eight benchmarks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Value
Group Size ( G )
8
Training Batch Size
128
PPO Mini-batch Size
32
KL Loss Type
Low-variance KL
KL Coefficient ( β )
1×10−3
Rollout Top- p
1.0
Appendix
Table 5: Key hyperparameters used.
Settings
MATH500
GPQA
BBH
Minerva
AIME24
AIME25
IFBench
IFEval
Avg.
GRPO
Seed 1
62.60
24.24
10.86
17.28
17.08
9.17
21.20
69.00
28.93
Seed 2
64.00
22.22
12.72
16.18
19.90
10.73
22.00
68.00
29.47
Seed 3
63.60
23.23
12.43
17.28
19.38
8.02
24.40
69.60
29.74
Mean ± Std.
63.40 ± 0.72
23.23 ± 1.01
12.00 ± 1.00
16.91 ± 0.64
18.79 ± 1.50
9.31 ± 1.36
22.53 ± 1.67
68.87 ± 0.81
29.38 ± 0.41
Negative Rollout
Appendix
Table 6: Robustness across three random seeds on Qwen3-1.7B. We report individual runs together with the mean and standard deviation across seeds.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. Consequently, an objective whose rewards are already near their upper bound can retain substantial influence when its rewards still vary within rollout groups. We introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This changes the relative contribution of each objective according to its observed reward headroom. We further derive an exact condition under which saturation aware reweighting reverses the sign of a rollout's aggregate advantage. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2% on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%, while in all settings maintaining the easier objectives near their already satisfied levels. Additional experiments characterize the accompanying reward tradeoffs and sensitivity to corrupted rewards.
Yixuan Wang, Yifei Chen, Haichao Zhang +6
University of Florida · UC San Diego · Northeastern University +3
Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.
Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.