Group Relative Policy Optimization (GRPO) has become the dominant reinforcement learning method for eliciting reasoning capabilities in LLMs using verifiable rewards. Still, it suffers from two critical issues that limit models' exploration: Gradient vanishing and Diversity collapse. When a training question is too easy or too hard, all sampled responses receive identical rewards, yielding no gradient signal. Meanwhile, the model tends to collapse its responses toward a single reasoning pattern rather than exploring diverse strategies. We propose Transformation-Augmented GRPO (TA-GRPO), a simple yet effective method that addresses both issues via question transformation. For each training question, we automatically create multiple problem-equivalent rephrased questions that alter wording, format, and information order while preserving the underlying intent. As these rephrasings shift the model's perceived difficulty, responses across the original question and its rephrasings receive mixed rewards and encode more diverse reasoning strategies within it. TA-GRPO jointly computes advantages over this expanded response group and aligns all importance sampling ratios to the original question, enabling the model to learn from a richer set of attempts. Experiments with four LLMs (Qwen3-1.7B/4B-Base and Llama-3.2-1B/3B-Instruct) show that TA-GRPO consistently improves pass@k on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves the average pass@32 of Qwen3-1.7B-Base and Qwen3-4B-Base by 4.97% and 4.35%, respectively, and matches the pass@32 of baselines trained with up to 2.5x more data.
Figures & tables
Figure 1: An example showing how different rephrasings of a question induce diverse reasoning strategies. Each column contains a question and the model’s response. The model adopts a distinct strategy for each rephrasing: standard interest formula (original), calculate quarterly interest rate (neighbor 1), calculate discount factor (neighbor 2), and calculate effective annual interest rate (neighbor N ). Despite these differences, all attempts arrive at the same final answer ($42,409), since the rephrasings preserve the underlying intent.
Figure 2: An illustration of our proposed TA-GRPO. Dashed arrows indicate rephrasing the original question. The blue arrow indicates updating the model.
Figure 3: The percentage of training questions for which responses within the group ( a ) and the expanded group ( a ) receive identical rewards. Base indicates the base models before training, while 25%, 50%, 75%, and 100% indicate checkpoints throughout the training process.
Figure 4: The density distribution of semantic diversity of responses within the group ( a ) and the expanded group ( a ). Base indicates the base models before training, while 25%, 50%, 75%, and 100% indicate checkpoints throughout the training process.
Figure 5: The pass@ k curves of models on six testing benchmarks.
Model
Method
AMC
OlympiadBench
AIME24
AIME25
Average
Minerva
GPQA-Diamond
Average
Qwen3-1.7B-Base
Base
62.70
60.04
32.54
29.13
46.10
49.64
61.10
55.37
GRPO
76.06
66.03
39.37
32.34
53.45
51.34
66.91
59.12
DAPO
77.49
67.31
40.38
33.21
54.60
52.18
67.94
60.06
TA-GRPO
83.74
70.45
47.42
37.14
59.69
53.07
76.52
64.80
Qwen3-4B-Base
Base
79.98
68.08
45.44
36.32
57.46
57.19
77.84
67.52
GRPO
84.16
70.55
46.47
42.11
60.82
58.25
78.96
68.60
Table 1: Detailed pass@32 of models on six testing benchmarks. Best results are in bold .
Figure 6: Detailed pass@32 of models on six testing benchmarks.
Figure 7: Average pass@32 of models across six testing benchmarks.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
max prompt length
512
max response length
4096
temperature
1.0
learning rate
1e-6
KL loss coefficient
0.0
clip ratio (lower)
0.2
clip ratio (upper)
0.2
Appendix
Table 2: Key Training Hyperparameters.
Model
Method
Training Times
Qwen3-1.7B-Base
GRPO
∼ 6 hours
DAPO
∼ 6 hours
(N+1)× GRPO
∼ 24 hours
(N+1)× DAPO
∼ 24 hours
TA-GRPO
∼ 24 hours
Qwen3-4B-Base
GRPO
∼ 12 hours
Appendix
Table 3: Training Times of TA-GRPO and Baselines.
Figure 8: Average pass@32 of models across six testing benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, where Group Relative Policy Optimization (GRPO) serves as the mainstream algorithm. We point out two understudied inefficiencies existing in GRPO. First, the fixed KL penalty coefficient overly restricts policy exploration at stages where the model requires significant deviation from the reference policy. Second, uniform sampling of training questions ignores that moderately difficult problems provide the most informative gradient signals for optimization. We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules. The Accuracy-Conditioned KL Scaling (AKL) dynamically adjusts KL regularization strength through a smooth nonlinear function of batch average accuracy, relaxing the penalty when the model underperforms and strengthening it when the model achieves good results. The Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at moderate accuracy around 0.5, focusing training on the model's learning frontier. We conduct extensive experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks. The results show EXPO steadily surpasses vanilla GRPO. It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model. The much larger performance gains on pass@32 compared with pass@1 demonstrate that EXPO effectively enlarges the model's exploration boundary under a fixed inference cost budget.
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney
Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
Junoh Park, Junseo Hwang, Wonguk Cho +1
Graduate School of Data Science, Seoul National University