Organizations: School of Intelligence Science and Technology, Peking University · Baidu Inc. · Department of Computer Science, University of Toronto · Institute of Automation, Chinese Academy of Sciences · Aerospace Information Research Institute, Chinese Academy of Sciences · Gaoling School of Artificial Intelligence, Renmin University of China
Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models (LLMs). However, existing RLVR methods often suffer from exploration inefficiency due to mismatches between problem difficulty and model capability: overly difficult problems hinder reasoning path discovery, while overly simple problems offer little learning signal. To address this, we first formalize the effect of problem difficulty by quantifying the relationship between loss descent magnitude and rollout accuracy. Building on this analysis, we propose SEELE, a supervision-aided RLVR framework that dynamically adjusts problem difficulty to lie within the high-performance region. SEELE augments each training sample by appending a hint (part of a full solution) for difficulty reduction. Unlike previous hint-based approaches, SEELE deliberately computes the hint length for each individual problem to achieve an optimal difficulty. The optimal hint length is determined via multi-round rollout sampling, where an item response theory model fits accuracy-hint pairs from previous rounds to predict the next-round hint. This instance-level, real-time difficulty adjustment aligns problem difficulty with the evolving model capability, thereby improving exploration efficiency. Experiments show that SEELE outperforms Group Relative Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +10.0 and +8.4 points, respectively, and exceeds the best prior supervision-aided approach by +3.8 points on average across six math reasoning benchmarks.
Figures & tables
Figure 1: Comparison between the direct rollout (blue) and hinted rollout (purple). The hint consists of the first few steps from an annotated solution. Adding a hint can simplify the problem and guide the LLM toward completing correct solutions.
Figure 2: Overview of SEELE. In each step, SEELE conducts m rollout rounds. In round i , an adaptive hint y1:l(i) is appended to the original problem, where y1:l(i) is a prefix of the full solution y with length l(i) , determined by the accuracy-hint model fϕ to achieve the target rollout accuracy a∗ . The output accuracies within each round are then used to update fϕ , enabling more accurate predictions in subsequent rounds. Finally, SEELE computes the advantage function over the outputs of all m rounds, which is used to update the policy.
Figure 3: Cases of the accuracy-hint curves and the 3PL fitted curve and parameters. We select 4 typical curves to demonstrate the expressive power of the 3PL model.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
Qwen2.5-Math-7B
71.6
63.2
25.7
32.0
14.1
45.2
42.0
69.5
24.7
17.7
37.3
+ SFT
89.5
74.6
35.7
37.9
12.4
52.2
50.4
77.6
37.4
47.9
54.3
+ GRPO
92.0
80.6
36.0
41.2
23.9
60.2
55.7
77.1
37.4
45.2
53.2
+ LUFFY
91.7
80.0
35.3
42.4
18.5
66.2
55.7
80.5
39.9
49.9
56.8
+ UFT
92.1
82.4
34.6
40.3
17.6
66.6
55.6
81.0
40.9
49.9
57.3
Table 1: Accuracy on math and general domain reasoning benchmarks.
Figure 4: Training dynamics of RL compared with baselines on Qwen2.5-3B. Left : training rewards; Middle : Response length; Right : averaged accuracy on math reasoning validation sets.
Figure 5: Accuracy within each rollout round during training. The red dotted line denotes the targeted accuracy. Left : Qwen2.5-1.5B; Right : Qwen2.5-3B.
Method
Math Avg
Gen Avg
h=1
24.0
38.7
h=2
31.5
40.7
h=3
31.7
40.5
h=4
31.5
38.6
Random
32.1
38.4
Linear
33.1
41.5
Table 2: Ablation on hint-length modeling using Qwen2.5-1.5B.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Phase
GRPO
SEELE
Rollout
2NLp+2mnNLr
2mNLp+2mnNLr
Old Prob.
2mnN(Lp+Lr)
2mnN(Lp+Lr)
Ref. Prob.
2mnN(Lp+Lr)
2mnN(Lp+Lr)
Update
6mnN(Lp+Lr)
6mnN(Lp+Lr)
Total
2NLp+2mnNLr+10mnN(Lp+Lr)
2mNLp+2mnNLr+10mnN(Lp+Lr)
Appendix
Table 3: FLOPs comparison with GRPO.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
Qwen2.5-3B
73.3
29.0
7.0
10.7
0.6
11.6
22.0
66.6
23.7
15.2
35.2
+ SFT
73.1
52.6
18.4
18.5
2.3
25.0
31.7
75.7
30.3
40.1
48.7
+ GRPO
78.5
45.8
17.3
15.0
1.7
23.9
30.4
75.6
36.9
40.5
51.0
+ LUFFY
81.1
56.6
21.0
21.3
2.7
27.0
35.0
78.4
32.8
41.5
50.9
+ UFT
84.8
62.6
22.1
25.9
5.8
41.6
40.5
79.4
35.4
42.9
52.6
Appendix
Table 4: Performance on math and general domain reasoning benchmarks using Qwen2.5-3B, LLaMA 3.2-3B, and Mathstral-7B-v0.1.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
DeepSeek-R1-Distill-Qwen-7B
89.5
72.6
25.7
37.9
17.3
53.3
49.4
69.7
34.8
30.7
45.1
+ GRPO
90.1
84.4
31.2
51.1
27.9
75.5
60.0
75.9
44.4
51.1
57.1
+ SEELE (Ours)
90.3
87.2
34.9
53.0
38.9
79.0
63.9
79.1
42.4
53.6
58.4
Qwen3-4B-Instruct-2507
94.1
86.6
35.7
49.0
29.4
75.2
61.7
90.2
52.0
60.0
67.4
+ SFT
83.5
65.2
30.1
28.3
7.2
41.0
42.6
89.5
45.5
60.2
65.1
Appendix
Table 5: Performance on math and general domain reasoning benchmarks using DeepSeek-R1-Distill-Qwen-7B and Qwen3-4B-Instruct-2507.
Method
Math Average
General Average
P@1
P@4
P@8
P@16
P@1
P@4
P@8
P@16
GRPO
55.7
61.8
64.2
66.3
53.2
57.2
58.6
63.1
UFT
55.6
60.7
63.2
65.3
57.3
64.1
66.0
67.1
Prefix-RFT
56.3
62.1
64.8
67.5
55.8
60.1
61.4
62.2
SEELE (Ours)
59.0
64.5
66.0
69.6
58.5
64.9
67.0
68.2
Appendix
Table 6: Pass@k comparison across math and general reasoning benchmarks using Qwen2.5-Math-7B. SEELE consistently achieves the best performance for all k values.
Figure 6: Performance for different target accuracies a∗ using Qwen2.5-1.5B.
Figure 7: Reward across the training steps for various target accuracies.
Multi-Round
Math Reasoning
General Domain Reasoning
Rollout Scheme
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
mn=8
m=2
76.0
50.8
16.2
18.2
3.0
28.1
32.1
65.2
26.3
28.1
39.9
mn=16
m=4
76.8
51.0
15.4
19.0
3.3
28.4
32.3
66.3
24.7
29.7
40.2
mn=24
m=3
75.1
53.4
14.7
20.0
4.6
31.8
33.3
67.9
21.9
29.8
39.9
m=4
77.1
54.4
16.9
19.9
3.0
32.1
33.9
66.6
24.7
33.2
41.5
m=6
77.6
53.8
15.8
18.5
2.9
30.5
33.2
68.9
23.8
33.0
41.9
Appendix
Table 7: Performance of different multi-round configurations using Qwen2.5-1.5B.
Figure 8: Accuracy with respect to the hinting rate for 100 training examples using Qwen2.5-3B.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement Learning with Verifiable Reward (RLVR) is empirically shown to notably enhance the reasoning performance of large language models (LLMs), particularly in mathematics and programming. However, the mechanistic role of Sample Difficulty in RLVR remains poorly understood. In this paper, we investigate RLVR through the lens of difficulty-wise and one-sample analysis. We find that sample difficulty has a non-monotonic effect on RLVR: easy and medium-difficulty problems yield the strongest and most stable reasoning improvements, whereas overly hard problems often provide weak learning signals, induce degenerate behaviors such as answer repetition or skipping necessary computation, and can ultimately degrade the model's pre-existing capabilities. Beyond the obverse of response, we further analyze the model's internal feature dynamics using Temporal Sparse Autoencoders (T-SAE). Easy problems mainly reinforce direct-answer and basic-computation features while suppressing deliberative-reasoning features; hard problems activate reasoning-related features but become useful only when successful trajectories are sampled; medium-difficulty problems provide a more balanced signal, strengthening both computation and multi-step reasoning features. Motivated by these findings, we propose difficulty-adaptive strategies for hard-sample utilization, using backward-reasoning reformulation and T-SAE-guided training signals to improve reward density and credit assignment during RLVR. Overall, our results identify sample difficulty as a key factor governing both the optimization dynamics and representation evolution of RLVR.
Yue Cheng, Jiajun Zhang, Xiaohui Gao +3
Beijing Jiaotong University · AntGroup · Northwestern Polytechnical University +2
Reinforcement learning (RL) has demonstrated potential for enhancing reasoning in large language models (LLMs). However, effective RL training, which requires medium-difficulty training samples, faces two fundamental challenges: Effective Data Scarcity and Dynamic Difficulty Shifts, where medium-difficulty samples are scarce and become trivial as models improve. Existing methods mitigate this scarcity to some extent by generating training samples. However, these approaches suffer from anchor-free generation, ignoring co-evolution, and difficulty mismatch. To address these issues, we propose D2Evo, a Dual Difficulty-aware self-Evolution RL framework. In each iteration, our method mines medium-difficulty anchors based on the current Solver's capability, trains the Questioner to generate diverse questions at appropriate difficulty levels, and jointly optimizes both components to enable progressive reasoning gains. Extensive experiments demonstrate that D2Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples, and exhibits strong generalization on general reasoning benchmarks.
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.