DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

Zhongwei Wan; Yun Shen; Zhihao Dou; Donghao Zhou; Yu Zhang; Xin Wang; Hui Shen; Jing Xiong

cs.LGFeb 23, 2026

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

Authors: Zhongwei Wan, Yun Shen, Zhihao Dou, Donghao Zhou, Yu Zhang, Xin Wang, Hui Shen, Jing Xiong,

Organizations: The Ohio State University, USA · Case Western Reserve University, USA · The Chinese University of Hong Kong, Hong Kong · Macquarie University, Australia ·

Abstract

Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.

Figures & tables

Panel of Figure 1 — Figure 1: ( Left ): global-to-local coupling for enhanced exploration during RL training. ( Right ): baseline exploration collapses to local suboptimal solutions, while DSDR promotes diverse trajectories that escape local optima and reach the correct solution space.

Figure 2: DSDR training pipeline for dual-scale exploration in RL. Correct-only global diversity promotes exploration across solution modes, while a global-to-local coupling mechanism allocates length-invariant local entropy regularization to distinctive correct trajectories. Both signals are integrated into policy updates to enable deep exploration without sacrificing correctness.

Qwen2.5-math-1.5B , Max Response Length = 4K tokens
Models	AIME24	AIME25	MATH500	Minerva	Olympiad	Average
Backbone	10.0 / 3.0	3.3 / 3.0	27.4 / 21.0	5.9 / 3.9	13.1 / 9.9	11.9 / 8.2
GRPO	16.7 / 15.2	3.5 / 8.5	57.8 / 58.3	14.0 / 15.0	24.0 / 22.9	23.2 / 24.0
DAPO	13.3 / 14.4	6.7 / 6.3	54.8 / 55.3	15.8 / 15.2	22.7 / 21.4	22.7 / 22.5
DSDR	20.0 / 18.8	6.7 / 10.2	59.0 / 57.5	18.0 / 18.0	23.1 / 23.7	25.4 / 25.6
Qwen3-1.7B , Max Response Length = 8K tokens

Table 1: Results on different reasoning benchmarks. We report Pass@1 and Avg@16 (

\%

) accuracy across different model scales. DSDR consistently outperforms Backbone, GRPO, and DAPO across most benchmarks and achieves the best average performance. Ablation results (w/o GD, w/o GC) show consistent performance drops, suggesting that both global diversity (GD) and global-to-local coupling (GC) regularization play important roles in DSDR.

Figure 3: Pass@k performance across five benchmarks for both Qwen3-1.7B and Qwen3-4B. The Base models serve as backbones. DSDR consistently outperforms both the Base models and DAPO across all values of

k

Figure 4: Training dynamics across methods conducted on Qwen3-1.7 model. From left to right, we report AIME2024 Avg@16, policy entropy, semantic-level diversity similarity, and formula-level diversity similarity. Results are shown for GRPO, DSDR, DSDR w/o GD, DSDR w/o GC, and DAPO.

Figure 5: We generate 32 test-time rollouts per problem on four benchmarks and evaluate response diversity using an LLM-as-a-Judge (1–10 scale). The figure reports diversity scores and corresponding pass@32 for DAPO and DSDR.

Figure 6: Hyperparameter sensitivity of DSDR on Qwen3-1.7B. Left: varying

\lambda_{\ell}

shows that overly large entropy regularization destabilizes training. Right:

\lambda_{d}=0.001

achieves the best and most stable Avg@16 performance on AIME2024/2025.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Training Settings
Hardware	8 $\times$ A100 GPUs (40GB)
Base models	Qwen2.5-Math-1.5B / Qwen3-1.7B, 4B
Training dataset	open-r1/DAPO-Math-17K-Processed
Max response length	4096 / 8192
Batch / mini-batch size	256 / 16
Rollout group size $n$	8

Appendix

Table 2: Summary of training details.

Evaluation settings
Max response length	4096 / 8192
Top- $p$ (eval)	0.9
Temperature (eval)	0.1 for Pass@1; 0.7 for Avg@16

Appendix

Table 3: Evaluation settings.

Figure 7: Training dynamics across methods conducted on Qwen3-1.7B model. From left to right, we show Avg@16 and Pass@16 for AIME2025, PG loss, and response length/clip ratio, comparing GRPO, DSDR, DSDR w/o GD, DSDR w/o GC, and DAPO.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning
Jun 9, 2026Jiangnan Xia, Yucheng Shi, Yu Yang +3Memorization in Language Models RL for Language Model Reasoning
University of Georgia · Tencent AI Lab · The Education University of Hong Kong +1
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

Oct 5, 2025J/K move · Enter open · S save

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Zishang Jiang, Jinyi Han, Tingyun Li, Xinyi Wang+6

School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University+1

RL for Language Model Reasoning LLM-Guided RL RL Exploration Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Abstract

Reinforcement Learning

Explore similar work

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning

Boosting LLM Exploration via Weak-Model Guidance in RLVR