Organizations: Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@k for large k. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as k scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
Figures & tables
Figure 1: Step-level entropy dynamics of Qwen2.5-7B conditioned on prefixes generated by Gemma-2-2B and Qwen2.5-7B itself, before and after GRPO training. Cross-model prefixes induce higher entropy in the early reasoning steps, suggesting broader exploration.
Figure 2: Overview of the proposed prefix-completion RLVR framework. Auxiliary small models first generate candidate reasoning trajectories, which are then truncated into partial prefixes. During RLVR training, the target model is trained to complete the remaining reasoning process conditioned on either the original question or the prefix-augmented input.
Method
pass@1
pass@2
pass@4
pass@8
pass@16
pass@32
pass@64
pass@128
Qwen2.5-7B
27.83
36.37
44.07
51.01
57.33
62.99
67.98
72.15
Qwen2.5-7B + GRPO
38.29
43.97
49.20
53.90
58.06
61.75
65.02
67.76
Qwen2.5-7B + LLaMA-3.2-1B prefix
38.15
43.53
48.52
53.11
57.48
61.75
65.75
69.06
Qwen2.5-7B + Gemma-2-2B prefix
39.01
44.60
49.52
54.03
58.41
62.66
66.78
70.71
Qwen2.5-Math-7B
26.45
36.83
46.22
53.77
59.58
64.29
68.52
72.71
Qwen2.5-Math-7B + GRPO
40.08
45.86
51.13
55.91
60.19
63.72
66.63
69.30
Table 1: Average pass@ k performance across all 6 mathematical reasoning benchmarks of different methods.
Figure 3: Training dynamics under different prefix probabilities p .
Method
pass@1
pass@128
p=0 (Vanilla GRPO)
38.29
67.76
Prefix Probability p
p=0.2
39.01
70.71
p=0.5
39.22
67.89
p=1.0
38.56
68.54
Truncation Strategy
Table 2: Ablation results on Qwen2.5-7B.
Figure 4: Fine-grained quality distribution of prefixes across different generator models.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Prompt for 7B models
Figure 6: Prompt for smaller models
Figure 7: Prompt for DeepSeek-v4-Flash prefix assessment.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Zishang Jiang, Jinyi Han, Tingyun Li +7
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement learning with verifiable rewards (RLVR) recently thrives in large language model (LLM) reasoning tasks. However, the reward sparsity and the long reasoning horizon make effective exploration challenging. In practice, this challenge manifests as the entropy collapse phenomenon, where RLVR improves single-rollout accuracy but fails to expand coverage on successful reasoning trajectories. Passive exploration techniques like entropy regularization tend to dismiss generation quality, resulting in noisy rollouts. In response to this issue, we propose an Information-Maximizing Augmented eXploration (IMAX) framework to train a pool of soft prefixes that reshapes the base model's prior over reasoning trajectories. Rather than relying on RL to incentivize exploration on top of the base model, each prefix acts as a trainable control knob that induces a distinct rollout distribution from the same backbone model. To encourage discovery of diverse and task-relevant reasoning behaviors, we derive an Information Maximization (InfoMax) reward to complement the verifiable rewards for RL training. IMAX is in general algorithm-agnostic and can be seamlessly integrated into existing RLVR pipelines. Experiment results have shown that across three backbone scales, IMAX consistently improves reasoning performance over standard RLVR, with gains up to 11.60% in Pass@4 and 10.57% in Avg@4.
Yifan Xu, Junren Chen, Yifan Chen
Hong Kong Baptist University · University of Maryland, College Park
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.