Organizations: School of Intelligence Science and Technology, Peking University · Baidu Inc. · Department of Computer Science, University of Toronto · Institute of Automation, Chinese Academy of Sciences · Aerospace Information Research Institute, Chinese Academy of Sciences · Gaoling School of Artificial Intelligence, Renmin University of China
Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models (LLMs). However, existing RLVR methods often suffer from exploration inefficiency due to mismatches between problem difficulty and model capability: overly difficult problems hinder reasoning path discovery, while overly simple problems offer little learning signal. To address this, we first formalize the effect of problem difficulty by quantifying the relationship between loss descent magnitude and rollout accuracy. Building on this analysis, we propose SEELE, a supervision-aided RLVR framework that dynamically adjusts problem difficulty to lie within the high-performance region. SEELE augments each training sample by appending a hint (part of a full solution) for difficulty reduction. Unlike previous hint-based approaches, SEELE deliberately computes the hint length for each individual problem to achieve an optimal difficulty. The optimal hint length is determined via multi-round rollout sampling, where an item response theory model fits accuracy-hint pairs from previous rounds to predict the next-round hint. This instance-level, real-time difficulty adjustment aligns problem difficulty with the evolving model capability, thereby improving exploration efficiency. Experiments show that SEELE outperforms Group Relative Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +10.0 and +8.4 points, respectively, and exceeds the best prior supervision-aided approach by +3.8 points on average across six math reasoning benchmarks.
Figures & tables
Figure 1: Comparison between the direct rollout (blue) and hinted rollout (purple). The hint consists of the first few steps from an annotated solution. Adding a hint can simplify the problem and guide the LLM toward completing correct solutions.
Figure 2: Overview of SEELE. In each step, SEELE conducts m rollout rounds. In round i , an adaptive hint y1:l(i) is appended to the original problem, where y1:l(i) is a prefix of the full solution y with length l(i) , determined by the accuracy-hint model fϕ to achieve the target rollout accuracy a∗ . The output accuracies within each round are then used to update fϕ , enabling more accurate predictions in subsequent rounds. Finally, SEELE computes the advantage function over the outputs of all m rounds, which is used to update the policy.
Figure 3: Cases of the accuracy-hint curves and the 3PL fitted curve and parameters. We select 4 typical curves to demonstrate the expressive power of the 3PL model.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
Qwen2.5-Math-7B
71.6
63.2
25.7
32.0
14.1
45.2
42.0
69.5
24.7
17.7
37.3
+ SFT
89.5
74.6
35.7
37.9
12.4
52.2
50.4
77.6
37.4
47.9
54.3
+ GRPO
92.0
80.6
36.0
41.2
23.9
60.2
55.7
77.1
37.4
45.2
53.2
+ LUFFY
91.7
80.0
35.3
42.4
18.5
66.2
55.7
80.5
39.9
49.9
56.8
+ UFT
92.1
82.4
34.6
40.3
17.6
66.6
55.6
81.0
40.9
49.9
57.3
Table 1: Accuracy on math and general domain reasoning benchmarks.
Figure 4: Training dynamics of RL compared with baselines on Qwen2.5-3B. Left : training rewards; Middle : Response length; Right : averaged accuracy on math reasoning validation sets.
Figure 5: Accuracy within each rollout round during training. The red dotted line denotes the targeted accuracy. Left : Qwen2.5-1.5B; Right : Qwen2.5-3B.
Method
Math Avg
Gen Avg
h=1
24.0
38.7
h=2
31.5
40.7
h=3
31.7
40.5
h=4
31.5
38.6
Random
32.1
38.4
Linear
33.1
41.5
Table 2: Ablation on hint-length modeling using Qwen2.5-1.5B.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Phase
GRPO
SEELE
Rollout
2NLp+2mnNLr
2mNLp+2mnNLr
Old Prob.
2mnN(Lp+Lr)
2mnN(Lp+Lr)
Ref. Prob.
2mnN(Lp+Lr)
2mnN(Lp+Lr)
Update
6mnN(Lp+Lr)
6mnN(Lp+Lr)
Total
2NLp+2mnNLr+10mnN(Lp+Lr)
2mNLp+2mnNLr+10mnN(Lp+Lr)
Appendix
Table 3: FLOPs comparison with GRPO.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
Qwen2.5-3B
73.3
29.0
7.0
10.7
0.6
11.6
22.0
66.6
23.7
15.2
35.2
+ SFT
73.1
52.6
18.4
18.5
2.3
25.0
31.7
75.7
30.3
40.1
48.7
+ GRPO
78.5
45.8
17.3
15.0
1.7
23.9
30.4
75.6
36.9
40.5
51.0
+ LUFFY
81.1
56.6
21.0
21.3
2.7
27.0
35.0
78.4
32.8
41.5
50.9
+ UFT
84.8
62.6
22.1
25.9
5.8
41.6
40.5
79.4
35.4
42.9
52.6
Appendix
Table 4: Performance on math and general domain reasoning benchmarks using Qwen2.5-3B, LLaMA 3.2-3B, and Mathstral-7B-v0.1.
Model
Math Reasoning
General Domain Reasoning
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
DeepSeek-R1-Distill-Qwen-7B
89.5
72.6
25.7
37.9
17.3
53.3
49.4
69.7
34.8
30.7
45.1
+ GRPO
90.1
84.4
31.2
51.1
27.9
75.5
60.0
75.9
44.4
51.1
57.1
+ SEELE (Ours)
90.3
87.2
34.9
53.0
38.9
79.0
63.9
79.1
42.4
53.6
58.4
Qwen3-4B-Instruct-2507
94.1
86.6
35.7
49.0
29.4
75.2
61.7
90.2
52.0
60.0
67.4
+ SFT
83.5
65.2
30.1
28.3
7.2
41.0
42.6
89.5
45.5
60.2
65.1
Appendix
Table 5: Performance on math and general domain reasoning benchmarks using DeepSeek-R1-Distill-Qwen-7B and Qwen3-4B-Instruct-2507.
Method
Math Average
General Average
P@1
P@4
P@8
P@16
P@1
P@4
P@8
P@16
GRPO
55.7
61.8
64.2
66.3
53.2
57.2
58.6
63.1
UFT
55.6
60.7
63.2
65.3
57.3
64.1
66.0
67.1
Prefix-RFT
56.3
62.1
64.8
67.5
55.8
60.1
61.4
62.2
SEELE (Ours)
59.0
64.5
66.0
69.6
58.5
64.9
67.0
68.2
Appendix
Table 6: Pass@k comparison across math and general reasoning benchmarks using Qwen2.5-Math-7B. SEELE consistently achieves the best performance for all k values.
Figure 6: Performance for different target accuracies a∗ using Qwen2.5-1.5B.
Figure 7: Reward across the training steps for various target accuracies.
Multi-Round
Math Reasoning
General Domain Reasoning
Rollout Scheme
GSM8K
MATH500
Minerva
Olympiad
AIME24
AMC23
Avg.
ARC-C
GPQA-D
MMLU-Pro
Avg.
mn=8
m=2
76.0
50.8
16.2
18.2
3.0
28.1
32.1
65.2
26.3
28.1
39.9
mn=16
m=4
76.8
51.0
15.4
19.0
3.3
28.4
32.3
66.3
24.7
29.7
40.2
mn=24
m=3
75.1
53.4
14.7
20.0
4.6
31.8
33.3
67.9
21.9
29.8
39.9
m=4
77.1
54.4
16.9
19.9
3.0
32.1
33.9
66.6
24.7
33.2
41.5
m=6
77.6
53.8
15.8
18.5
2.9
30.5
33.2
68.9
23.8
33.0
41.9
Appendix
Table 7: Performance of different multi-round configurations using Qwen2.5-1.5B.
Figure 8: Accuracy with respect to the hinting rate for 100 training examples using Qwen2.5-3B.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.