cs.LGJan 30, 2026

Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models

Authors: Khiem Le, Phuc Nguyen, Youssef Mroueh, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla

Organizations: University of Notre Dame · IBM Research · Meta AI · Florida State University

Abstract

Group Relative Policy Optimization (GRPO) has become the dominant reinforcement learning method for eliciting reasoning capabilities in LLMs using verifiable rewards. Still, it suffers from two critical issues that limit models' exploration: Gradient vanishing and Diversity collapse. When a training question is too easy or too hard, all sampled responses receive identical rewards, yielding no gradient signal. Meanwhile, the model tends to collapse its responses toward a single reasoning pattern rather than exploring diverse strategies. We propose Transformation-Augmented GRPO (TA-GRPO), a simple yet effective method that addresses both issues via question transformation. For each training question, we automatically create multiple problem-equivalent rephrased questions that alter wording, format, and information order while preserving the underlying intent. As these rephrasings shift the model's perceived difficulty, responses across the original question and its rephrasings receive mixed rewards and encode more diverse reasoning strategies within it. TA-GRPO jointly computes advantages over this expanded response group and aligns all importance sampling ratios to the original question, enabling the model to learn from a richer set of attempts. Experiments with four LLMs (Qwen3-1.7B/4B-Base and Llama-3.2-1B/3B-Instruct) show that TA-GRPO consistently improves pass@kk on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves the average pass@32 of Qwen3-1.7B-Base and Qwen3-4B-Base by 4.97% and 4.35%, respectively, and matches the pass@32 of baselines trained with up to 2.5x more data.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning

    Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model ReasoningPolicy Optimization

  2. ReCo: Reweighting GRPO Against Distributional Concentration

    Jul 29, 2026Junoh Park, Junseo Hwang, Wonguk Cho +1Reasoning DiversityRL for Language Model Reasoning