Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
Organizations: University of Notre Dame · IBM Research · Meta AI · Florida State University
Abstract
Group Relative Policy Optimization (GRPO) has become the dominant reinforcement learning method for eliciting reasoning capabilities in LLMs using verifiable rewards. Still, it suffers from two critical issues that limit models' exploration: Gradient vanishing and Diversity collapse. When a training question is too easy or too hard, all sampled responses receive identical rewards, yielding no gradient signal. Meanwhile, the model tends to collapse its responses toward a single reasoning pattern rather than exploring diverse strategies. We propose Transformation-Augmented GRPO (TA-GRPO), a simple yet effective method that addresses both issues via question transformation. For each training question, we automatically create multiple problem-equivalent rephrased questions that alter wording, format, and information order while preserving the underlying intent. As these rephrasings shift the model's perceived difficulty, responses across the original question and its rephrasings receive mixed rewards and encode more diverse reasoning strategies within it. TA-GRPO jointly computes advantages over this expanded response group and aligns all importance sampling ratios to the original question, enabling the model to learn from a richer set of attempts. Experiments with four LLMs (Qwen3-1.7B/4B-Base and Llama-3.2-1B/3B-Instruct) show that TA-GRPO consistently improves pass@ on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves the average pass@32 of Qwen3-1.7B-Base and Qwen3-4B-Base by 4.97% and 4.35%, respectively, and matches the pass@32 of baselines trained with up to 2.5x more data.
Figures & tables
| Model | Method | AMC | OlympiadBench | AIME24 | AIME25 | Average | Minerva | GPQA-Diamond | Average |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B-Base | Base | 62.70 | 60.04 | 32.54 | 29.13 | 46.10 | 49.64 | 61.10 | 55.37 |
| GRPO | 76.06 | 66.03 | 39.37 | 32.34 | 53.45 | 51.34 | 66.91 | 59.12 | |
| DAPO | 77.49 | 67.31 | 40.38 | 33.21 | 54.60 | 52.18 | 67.94 | 60.06 | |
| TA-GRPO | 83.74 | 70.45 | 47.42 | 37.14 | 59.69 | 53.07 | 76.52 | 64.80 | |
| Qwen3-4B-Base | Base | 79.98 | 68.08 | 45.44 | 36.32 | 57.46 | 57.19 | 77.84 | 67.52 |
| GRPO | 84.16 | 70.55 | 46.47 | 42.11 | 60.82 | 58.25 | 78.96 | 68.60 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| max prompt length | 512 |
| max response length | 4096 |
| temperature | 1.0 |
| learning rate | 1e-6 |
| KL loss coefficient | 0.0 |
| clip ratio (lower) | 0.2 |
| clip ratio (upper) | 0.2 |
| Model | Method | Training Times |
|---|---|---|
| Qwen3-1.7B-Base | GRPO | 6 hours |
| DAPO | 6 hours | |
| GRPO | 24 hours | |
| DAPO | 24 hours | |
| TA-GRPO | 24 hours | |
| Qwen3-4B-Base | GRPO | 12 hours |
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,158 papers