Organizations: The Ohio State University, USA · Case Western Reserve University, USA · The Chinese University of Hong Kong, Hong Kong · Macquarie University, Australia · University of Michigan, USA · The University of Hong Kong, Hong Kong · University College London, UK
Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.
Figures & tables
Figure 1: ( Left ): global-to-local coupling for enhanced exploration during RL training. ( Right ): baseline exploration collapses to local suboptimal solutions, while DSDR promotes diverse trajectories that escape local optima and reach the correct solution space.
Figure 2: DSDR training pipeline for dual-scale exploration in RL. Correct-only global diversity promotes exploration across solution modes, while a global-to-local coupling mechanism allocates length-invariant local entropy regularization to distinctive correct trajectories. Both signals are integrated into policy updates to enable deep exploration without sacrificing correctness.
Models
AIME24
AIME25
MATH500
Minerva
Olympiad
Average
Qwen2.5-math-1.5B , Max Response Length = 4K tokens
Backbone
10.0 / 3.0
3.3 / 3.0
27.4 / 21.0
5.9 / 3.9
13.1 / 9.9
11.9 / 8.2
GRPO
16.7 / 15.2
3.5 / 8.5
57.8 / 58.3
14.0 / 15.0
24.0 / 22.9
23.2 / 24.0
DAPO
13.3 / 14.4
6.7 / 6.3
54.8 / 55.3
15.8 / 15.2
22.7 / 21.4
22.7 / 22.5
DSDR
20.0 / 18.8
6.7 / 10.2
59.0 / 57.5
18.0 / 18.0
23.1 / 23.7
25.4 / 25.6
Qwen3-1.7B , Max Response Length = 8K tokens
Table 1: Results on different reasoning benchmarks. We report Pass@1 and Avg@16 ( % ) accuracy across different model scales. DSDR consistently outperforms Backbone, GRPO, and DAPO across most benchmarks and achieves the best average performance. Ablation results (w/o GD, w/o GC) show consistent performance drops, suggesting that both global diversity (GD) and global-to-local coupling (GC) regularization play important roles in DSDR.
Figure 3: Pass@k performance across five benchmarks for both Qwen3-1.7B and Qwen3-4B. The Base models serve as backbones. DSDR consistently outperforms both the Base models and DAPO across all values of k ..
Figure 4: Training dynamics across methods conducted on Qwen3-1.7 model. From left to right, we report AIME2024 Avg@16, policy entropy, semantic-level diversity similarity, and formula-level diversity similarity. Results are shown for GRPO, DSDR, DSDR w/o GD, DSDR w/o GC, and DAPO.
Figure 5: We generate 32 test-time rollouts per problem on four benchmarks and evaluate response diversity using an LLM-as-a-Judge (1–10 scale). The figure reports diversity scores and corresponding pass@32 for DAPO and DSDR.
Figure 6: Hyperparameter sensitivity of DSDR on Qwen3-1.7B. Left: varying λℓ shows that overly large entropy regularization destabilizes training. Right: λd=0.001 achieves the best and most stable Avg@16 performance on AIME2024/2025.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Training Settings
Hardware
8 × A100 GPUs (40GB)
Base models
Qwen2.5-Math-1.5B / Qwen3-1.7B, 4B
Training dataset
open-r1/DAPO-Math-17K-Processed
Max response length
4096 / 8192
Batch / mini-batch size
256 / 16
Rollout group size n
8
Appendix
Table 2: Summary of training details.
Evaluation settings
Max response length
4096 / 8192
Top- p (eval)
0.9
Temperature (eval)
0.1 for Pass@1; 0.7 for Avg@16
Appendix
Table 3: Evaluation settings.
Figure 7: Training dynamics across methods conducted on Qwen3-1.7B model. From left to right, we show Avg@16 and Pass@16 for AIME2025, PG loss, and response length/clip ratio, comparing GRPO, DSDR, DSDR w/o GD, DSDR w/o GC, and DAPO.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Zishang Jiang, Jinyi Han, Tingyun Li +7
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement learning has become a key paradigm for eliciting reasoning abilities in large language models, where exploration is crucial for discovering effective solution trajectories. Existing exploration methods typically encourage diversity in semantic or gradient spaces, without distinguishing what drives this diversity. A trajectory may appear novel because it follows a new reasoning process, or because it varies memorized patterns and shortcuts. Rewarding both cases equally may steer exploration toward memorization rather than genuine reasoning improvement. In this paper, we propose DiRL, a Direction-Aware Reinforcement Learning framework that anchors exploration to an internal reasoning-memorization direction of the policy. Specifically, DiRL extracts this direction from model representations, constructs direction-weighted gradient features to characterize rollout updates, and shapes rewards to amplify reasoning-aligned exploration while suppressing memorization-aligned variations. DiRL integrates seamlessly into standard Group Relative Policy Optimization (GRPO). Extensive experiments on mathematical and general reasoning benchmarks demonstrate the effectiveness of DiRL, showing significant improvements over various existing exploration methods.
Jiangnan Xia, Yucheng Shi, Yu Yang +3
University of Georgia · Tencent AI Lab · The Education University of Hong Kong +1
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@k for large k. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as k scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
Xingyu Shen, Huishuai Zhang, Peng Li +2
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies