Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: https://github.com/on1262/LEAF
Figures & tables
Random sampling
SDPO
SCoRe
RLVR
Ours
Rich feedback
✗
✓
✗
✗
✓
Inference adaptation
✗
✗
✓
✗
✓
Single rollout
✓
✗
✗
✗
✓
Figure 1: Properties of exploration methods as evaluated here. Rich feedback : uses textual feedback such as test output, not only the scalar reward. Inference adaptation : the rollout context changes with earlier attempts on the same problem, without a parameter update. Single rollout : one verified answer per problem per step.
Figure 2: Overview of LEAF. Rollout : each problem keeps a memory of its last response, reward and feedback, which forms the context of the next attempt. The memory is updated after each trial. Policy update : after all rollout steps are finished, the teacher conditioned on a successful context teaches the student without context through self-distillation.
Figure 3: Cumulative pass rate on four benchmarks against (a) verifier calls per problem, (b) cumulative compute in PFLOPs (log scale) and (c) cumulative GPU time. Both axes are averaged over the two models at each step.
Qwen3-8B
Olmo-3-7B-Inst.
Method
Pass
GPU h
PFLOPs
Pass
GPU h
PFLOPs
Random Sampling
77.44
0.66
625
78.15
3.42
676
GRPO
77.78
4.43
2,494
78.00
9.58
2,885
SDPO
77.71
5.06
3,277
78.31
10.46
3,843
SCoRe
78.65
5.09
3,058
82.73
12.33
4,401
SEED
77.96
7.87
4,580
77.47
9.43
3,796
Table 1: Pass rate (%) averaged over the four benchmarks and summed exploration cost. Refer to Appendix D for results by seed and benchmark.
Qwen3-8B
Olmo-3-7B-Instruct
Method
LCB ↑
MATH ↑
SciKE ↑
Overall ↑
GPU h ↓
EFLOPs ↓
LCB ↑
MATH ↑
SciKE ↑
Overall ↑
GPU h ↓
EFLOPs ↓
Base
19.78
84.08
74.40
59.42
–
–
21.25
89.12
69.25
59.87
–
–
GRPO
26.02
86.53
76.75
63.10
5.52
3.71
21.12
89.33
70.83
60.43
14.06
4.52
SDPO
25.64
81.65
72.96
60.08
5.82
4.35
23.66
82.38
66.98
57.68
14.23
5.93
SCoRe
22.71
84.32
75.17
60.73
5.57
4.17
20.48
88.95
68.38
59.27
17.71
6.33
SEED
25.89
85.58
76.83
62.77
6.72
4.88
22.33
89.10
72.10
61.18
14.56
5.94
Table 2: In-distribution generalization results. Avg@4 (%) on test sets. LCB: LiveCodeBench v6. MATH: MATH-500. SciKE: SciKnowEval. GPU h and EFLOPs ( 1018 FLOPs) sum the three training runs. For LEAF and SFT they include the exploration rollout that collects the training data, and for SFT also rewriting and verifying the solutions. Refer to Appendix F for results by run seed and training cost by stage.
Figure 4: Exploration ablations on key components. Final pass rate against exploration GPU time, averaged over the two models. Our method in main experiments is denoted as red star.
(a) Exploration
(b) Generalization
Method
Length
Gain (pp.)
Length
Gain (pp.)
GRPO
1.00
0.09
1.14
2.12
SDPO
0.99
0.21
0.98
− 0.77
SCoRe
1.27
2.89
1.01
0.35
SEED
1.00
0.31
1.11
2.33
SFT
–
–
0.93
− 1.46
Table 3: Response length and accuracy gain over the reference. Length: ratio of the mean response length to that of the reference. Gain: accuracy gain over the reference in pp.
Qwen3-8B
Olmo-3-7B-Inst.
Method
Pass
GPU h
Pass
GPU h
Random Sampling (ref.)
77.06
0.64
77.81
3.39
Ours (no update)
80.94
1.47
84.49
6.30
Ours + update every step
80.48
5.55
85.16
13.97
Ours + update every 2 steps
80.80
4.38
84.14
11.40
Ours + update every 5 steps
80.53
3.40
84.25
9.70
Table 4: Final cumulative pass rate (%, averaged over the four benchmarks) and exploration GPU hours.
Qwen3-8B
Olmo-3-7B-Inst.
Traces
LCB
MATH
SciKE
LCB
MATH
SciKE
Successful
82.98
90.86
83.23
71.03
96.16
67.93
Failure
39.82
60.30
67.07
44.76
79.48
50.13
Random
69.30
88.54
80.16
69.24
93.76
66.80
Table 5: Teacher accuracy (%) on the training problems under different rollout traces.
Qwen3-8B
Olmo-3-7B-Inst.
Method
Overall
Trunc.
Overall
Trunc.
Base
59.41
0.02
59.84
2.04
Ours (reverse KL)
64.92
2.25
62.71
2.45
Ours w/ forward KL
63.80
1.77
60.93
2.02
Ours w/ JSD
59.69
0.10
62.04
1.83
Table 6: Ablations on three loss types. Overall test avg@4 (%) and truncated test responses (%, mean over the three test sets).
Qwen3-8B
Olmo-3-7B-Instruct
Benchmark
Wrong
Repeat (%)
Wrong
Repeat (%)
MATH-500
1,912
45.7
1,309
47.4
LCB
2,480
61.8
2,452
40.5
MBPP+
1,278
77.3
1,675
78.0
Table 7: Repeated wrong answers among the 24 independent Random Sampling responses per problem. Wrong is the total number of wrong responses over all problems. Repeat is the fraction of wrong responses whose error signature equals that of an earlier wrong response to the same problem. Correctness on LCB and MBPP+ is judged on public tests.
Model
Benchmark
Comparison
Problems
Repeat (%)
Δ Repeat (pp)
Fix (%)
Δ Fix (pp)
Qwen3-8B
MATH-500
w/o feedback
143
52.5 / 65.0
−12.5 ( −17.9 , −7.4 )
29.5 / 19.8
+9.7 ( +4.2 , +15.1 )
w/o last response
148
51.6 / 12.4
+39.2 ( +34.1 , +44.4 )
30.9 / 53.0
−22.2 ( −28.0 , −16.7 )
LCB
w/o feedback
115
38.3 / 67.6
−29.4 ( −35.8 , −23.3 )
16.6 / 3.5
+13.0 ( +8.2 , +18.1 )
w/o last response
117
38.2 / 23.8
+14.4 ( +9.6 , +18.9 )
17.3 / 16.0
+1.3 ( −1.7 , +4.4 )
MBPP+
w/o feedback
114
36.5 / 79.2
−42.7 ( −49.0 , −36.0 )
54.6 / 16.7
+37.9 ( +30.9 , +44.9 )
w/o last response
126
36.0 / 27.3
+8.7 ( +2.8 , +15.1 )
55.4 / 60.5
−5.1 ( −11.9 , +1.2 )
Table 8: Repeat and fix rates of LEAF compared with two memory ablations over the first 24 exploration steps. Repeat and Fix entries are listed as LEAF / comparison, averaged per problem over transitions and then over the problems shared by both conditions (Problems). Δ is LEAF minus the comparison in percentage points, with the 95% bootstrap confidence interval over problems in parentheses.
LEAF
LEAF w/o feedback
Model
Benchmark
Repeat (%)
Base (%)
Ratio
Repeat (%)
Base (%)
Ratio
Qwen3-8B
MATH-500
60.9
16.6
3.67
73.4
17.3
4.24
LCB
41.8
23.3
1.79
66.8
30.2
2.21
MBPP+
55.4
40.9
1.35
85.0
38.1
2.23
Olmo-3-7B-Instruct
MATH-500
56.0
21.0
2.67
65.0
23.8
2.73
LCB
27.4
9.7
2.84
47.1
10.5
4.47
Table 9: Repeat rates relative to independent sampling, computed over all transitions of each condition together. Base is the fraction of the 24 Random Sampling responses to the same problem that are wrong with the same error signature as the previous answer, averaged over transitions. Ratio is the number of repeats divided by the sum of base frequencies.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Random Sampling
GRPO
SDPO
SCoRe
SEED
LEAF
Verifier calls / problem
24
24
24
24
24
24
GPU-hour cap / benchmark
5
5
5
5
5
5
Parameter updates
✗
✓
✓
✓
✓
✗
Questions / update
–
32
32
32
32
–
Responses / question
1 per step
8
8
4×2 turns
8
1 verified per step (+1 draft at restart)
Updates
–
47/13/36/38
47/13/36/38
47/13/36/38
8/2/6/6 (collect) + 39/11/30/32 (RL)
–
Appendix
Table 10: Exploration hyperparameters for all methods. The response limit is 16,384 tokens for Olmo-3-7B-Instruct on LCB, and SCoRe limits the two turns to 16,896 tokens in total. All methods with parameter updates use AdamW with weight decay 0.01, a maximum gradient norm of 1.0, and a constant learning rate. Dashes mark settings that do not apply or are not set separately for that method.
MATH-500
LCB
MBPP+
SciKnowEval
Overall
Model
Method
Acc.
Pass
Acc.
Pass
Acc.
Pass
Acc.
Pass
Acc.
Pass
Qwen3-8B
Random Sampling
84.14
97.33
20.14
41.48
72.71
85.80
74.83
85.17
62.96
77.44
GRPO
84.01
96.87
19.99
43.00
73.15
86.07
74.83
85.17
63.00
77.78
SDPO
83.97
97.27
19.77
42.75
72.98
86.07
74.76
84.75
62.87
77.71
SCoRe
84.45
96.13
20.42
41.98
72.16
85.89
76.35
90.58
63.34
78.65
SEED
84.04
97.33
19.72
43.77
72.96
85.89
74.85
84.83
62.89
77.96
Appendix
Table 11: Exploration results by benchmark, averaged over three run seeds (%). Acc. is the average accuracy over all answers up to the cutoff step, and Pass is the cumulative pass rate at that step. Overall is the macro average over the four benchmarks. The cutoff step of a run is the last step reached by all four benchmarks, as in Table 1 . It is step 16 for SEED on Olmo-3-7B-Instruct at seeds 1 and 2 and step 24 otherwise.
Pass (%) by seed
Mean over 3 seeds
Model
Method
Seed 1
Seed 2
Seed 3
Pass (%)
GPU h
PFLOPs
Qwen3-8B
Random Sampling
77.06
77.53
77.74
77.44±0.34
0.66±0.03
625±1
GRPO
78.18
76.97
78.18
77.78±0.70
4.43±0.09
2,494±3
SDPO
78.35
76.98
77.79
77.71±0.69
5.06±0.10
3,277±5
SCoRe
78.52
78.85
78.57
78.65±0.18
5.09±0.06
3,058±5
SEED
77.87
77.73
78.26
77.96±0.27
7.87±0.16
4,580±8
Appendix
Table 12: Exploration pass rate (%) by run seed, macro-averaged over MATH-500, LCB, MBPP+, and SciKnowEval, with the summed exploration cost underlying Table 1 . The last three columns are means ± sample standard deviations over 3 seeds. † The MATH-500 cell stopped at 16 steps, so the values are taken at step 16.
Model
Method
MATH-500
LCB
MBPP+
SciKnowEval
Total
Qwen3-8B
Random Sampling
0.30 / 199 / 12.1
0.09 / 73 / 4.4
0.10 / 88 / 5.4
0.17 / 265 / 16.2
0.66 / 625 / 38.1
GRPO
1.82 / 998 / 12.2
1.01 / 362 / 4.4
0.67 / 441 / 5.4
0.92 / 693 / 7.3
4.43 / 2,494 / 29.3
SDPO
2.04 / 1,351 / 12.0
1.10 / 477 / 4.4
0.84 / 584 / 5.4
1.08 / 865 / 7.4
5.06 / 3,277 / 29.2
SCoRe
2.07 / 1,152 / 10.8
1.03 / 413 / 3.8
0.85 / 570 / 5.3
1.14 / 923 / 6.6
5.09 / 3,058 / 26.4
SEED
3.20 / 1,889 / 15.0
1.28 / 665 / 5.4
1.53 / 869 / 6.7
1.88 / 1,157 / 8.7
7.87 / 4,580 / 35.7
LEAF
0.48 / 350 / 21.4
0.46 / 140 / 8.5
0.22 / 172 / 10.5
0.31 / 381 / 23.3
1.47 / 1,043 / 63.7
Appendix
Table 13: Exploration cost by benchmark, averaged over three run seeds. Each cell lists GPU h / PFLOPs / generated tokens (M), accumulated up to the cutoff step defined in Table 11 . Total is the sum over the four benchmarks.
x -axis
Seed 1
Seed 2
Seed 3
Mean
Mean curve
Verifier calls
1.97
2.20
1.63
1.94±0.29
1.90
PFLOPs
6.19
6.91
5.25
6.12±0.83
5.99
GPU time
3.85
4.10
3.33
3.76±0.40
3.70
Appendix
Table 14: Speedup of LEAF over SCoRe ( × ) on the two-model average curves of Figure 3 . We report the mean ± standard deviation of the per-seed ratios over 3 seeds. Mean curve is the ratio on the curve averaged over the three seeds.
Qwen3-8B
Olmo-3-7B-Instruct
Training set
Problems
∣{x:C+(x)=∅}∣
(c+,y+) pairs
∣{x:C+(x)=∅}∣
(c+,y+) pairs
LCB (v2–v5)
478
303.0
3,965
335.3
3,921
MATH (Level 5)
500
432.0
6,895
467.3
8,567
SciKnowEval (L5)
400
335.3
3,694
377.3
4,407
Appendix
Table 15: Training sets for in-distribution generalization and the data collected by the K=20 step exploration rollout of LEAF, averaged over three run seeds. ∣{x:C+(x)=∅}∣ is the number of problems with at least one successful context, which is also the number of problems used in the policy update. (c+,y+) pairs counts the distinct correct, untruncated answers with their contexts, summed over problems.
Model
Training set
Pairs
Kept
Dropped
Kept per round (1/2/3/4/5)
Length
Trunc. (%)
GPU h
Qwen3-8B
LCB
3,965
3,959
6
3,940 / 18 / 1 / 0 / 0
349
0.00
0.44
MATH
6,895
6,895
0
6,889 / 5 / 1 / 0 / 0
737
0.01
0.37
SciKnowEval
3,694
3,588
106
3,415 / 96 / 41 / 22 / 15
1,031
0.03
0.39
Olmo-3-7B-Instruct
LCB
3,921
3,899
22
3,823 / 57 / 12 / 5 / 2
404
0.00
0.58
MATH
8,567
8,566
1
8,562 / 2 / 0 / 1 / 0
597
0.00
0.95
SciKnowEval
4,407
3,964
443
3,031 / 484 / 227 / 136 / 86
899
3.96
1.49
Appendix
Table 16: Statistics of the rewritten solutions y∗ used by the SFT baseline, averaged over three run seeds and rounded. Pairs is the number of (c+,y+) pairs. Kept and Dropped count pairs whose rewrite did or did not pass the verifier within 5 rounds. Kept per round gives the number of pairs first accepted in each round. Length is the mean number of tokens of the kept rewrites. Trunc. is the fraction of truncated rewrite generations over all rounds. GPU h covers rewriting and verification.
Model
Method
LCB
MATH
SciKE
Overall
Qwen3-8B
Base
19.08 / 18.89 / 21.37
84.45 / 83.60 / 84.20
74.69 / 74.00 / 74.50
59.41 / 58.83 / 60.02
GRPO
25.38 / 25.95 / 26.72
86.35 / 86.90 / 86.35
75.62 / 77.25 / 77.38
62.45 / 63.37 / 63.48
SDPO
25.57 / 25.57 / 25.76
82.00 / 81.40 / 81.55
73.56 / 72.12 / 73.19
60.38 / 59.70 / 60.17
SCoRe
22.52 / 22.52 / 23.09
84.20 / 84.70 / 84.05
75.31 / 75.06 / 75.12
60.68 / 60.76 / 60.76
SEED
26.34 / 25.00 / 26.34
86.20 / 84.55 / 86.00
76.75 / 77.19 / 76.56
63.10 / 62.25 / 62.97
SFT
22.52 / 23.47 / 20.80
77.40 / 77.00 / 76.15
76.00 / 75.50 / 74.62
58.64 / 58.66 / 57.19
Appendix
Table 17: Test avg@4 results by run seed (%). Values are ordered by run seed 1 / 2 / 3. Overall is the mean of LCB, MATH and SciKE for the same run seed.
Model
Method
LCB
MATH
SciKE
Overall
Qwen3-8B
Base
19.78 / 31.55
84.08 / 93.53
74.40 / 81.00
59.42
GRPO
26.02 / 35.62
86.53 / 94.33
76.75 / 82.33
63.10
SDPO
25.64 / 34.10
81.65 / 91.40
72.96 / 80.58
60.08
SCoRe
22.71 / 32.82
84.32 / 93.60
75.17 / 81.92
60.73
SEED
25.89 / 35.88
85.58 / 93.87
76.83 / 82.42
62.77
SFT
22.26 / 33.08
76.85 / 88.87
75.38 / 84.83
58.16
Appendix
Table 18: Test avg@4 / pass@4 (%), averaged over three run seeds. Overall is the mean avg@4 over LCB, MATH and SciKE.
GPU h
EFLOPs
Model
Method
LCB
MATH
SciKE
Train
Rollout
Rewrite
Total
Train
Rollout
Rewrite
Total
Qwen3-8B
GRPO
1.78
2.42
1.31
5.52
–
–
5.52
3.71
–
–
3.71
SDPO
1.84
2.03
1.96
5.82
–
–
5.82
4.35
–
–
4.35
SCoRe
1.69
2.40
1.47
5.57
–
–
5.57
4.17
–
–
4.17
SEED
2.21
2.79
1.71
6.72
–
–
6.72
4.88
–
–
4.88
SFT
0.16
0.17
0.22
0.55
1.58
1.19
3.32
0.39
1.36
0.69
2.44
Appendix
Table 19: Training cost by training set and stage, averaged over three run seeds. LCB, MATH and SciKE give the training GPU h on each training set, and Train is their sum. Rollout is the exploration rollout that collects the training data of LEAF and SFT, and Rewrite is the generation and verification of the SFT targets. Total GPU h and Total EFLOPs ( 1018 FLOPs) equal the values in Table 2 . – marks a stage that the method does not use.
Qwen3-8B
Olmo-3-7B-Instruct
Method
MATH-500
LCB
MBPP+
SciKE
MATH-500
LCB
MBPP+
SciKE
Random Sampling
883
841
395
344
1,175
2,821
483
207
GRPO
886
839
395
345
1,175
2,792
478
206
SDPO
871
835
391
353
1,171
2,767
471
204
SCoRe
802
900
455
395
1,310
3,266
853
385
SEED
885
836
392
346
1,159
2,829
485
209
Appendix
Table 20: Mean response tokens per graded answer during exploration, averaged over three run seeds.
Qwen3-8B
Olmo-3-7B-Instruct
Method
LCB
MATH-500
SciKE
LCB
MATH-500
SciKE
Base
843 / 0.00
879 / 0.12
341 / 0.00
2,781 / 2.61
1,166 / 2.95
205 / 0.15
GRPO
936 / 0.00
1,001 / 0.35
412 / 0.00
3,049 / 2.54
1,201 / 3.10
254 / 0.29
SDPO
808 / 0.00
678 / 0.00
471 / 1.40
3,674 / 2.42
747 / 1.00
171 / 0.02
SCoRe
849 / 0.00
878 / 0.10
350 / 0.04
2,765 / 1.72
1,190 / 3.23
205 / 0.13
SEED
828 / 0.06
925 / 0.12
394 / 0.02
2,677 / 1.27
1,219 / 3.20
304 / 0.90
Appendix
Table 21: Test response length after the policy update. Each cell lists the mean completion tokens and the truncation rate (%), both averaged over three run seeds. Means include truncated responses.
Model
Variant
MATH-500
LCB
MBPP+
SciKE
Total
Qwen3-8B
LEAF
0.48 / 350 / 21.4
0.46 / 140 / 8.6
0.22 / 173 / 10.6
0.32 / 380 / 23.2
1.47 / 1,044 / 63.7
w/o last response
0.55 / 280 / 17.1
0.47 / 108 / 6.6
0.17 / 139 / 8.5
0.27 / 326 / 19.9
1.47 / 854 / 52.1
w/o feedback
0.52 / 371 / 22.6
0.49 / 140 / 8.5
0.20 / 194 / 11.8
0.35 / 434 / 26.5
1.56 / 1,139 / 69.5
Restart every 2
0.63 / 421 / 25.7
0.46 / 164 / 10.0
0.25 / 213 / 13.0
0.33 / 419 / 25.6
1.68 / 1,217 / 74.3
Restart every 10
0.43 / 331 / 20.2
0.50 / 134 / 8.2
0.21 / 160 / 9.8
0.29 / 367 / 22.4
1.43 / 992 / 60.6
No restart
0.39 / 314 / 19.2
0.52 / 124 / 7.5
0.20 / 145 / 8.9
0.25 / 351 / 21.4
1.36 / 934 / 57.0
Appendix
Table 22: Exploration cost of the component ablations. Each cell lists GPU hours, PFLOPs and generated tokens (millions), cumulative at the last step of the benchmark. Total is the sum over the four benchmarks.
Model
Variant
MATH-500
LCB
MBPP+
SciKE
Overall
Qwen3-8B
LEAF
87.12 / 97.20
29.48 / 47.33
74.17 / 86.24
80.52 / 93.00
67.82 / 80.94
w/o last response
84.07 / 98.00
21.56 / 46.56
71.88 / 86.24
77.02 / 88.00
63.63 / 79.70
w/o feedback
85.92 / 95.60
22.30 / 38.93
71.10 / 83.60
80.67 / 93.25
65.00 / 77.84
Restart every 2
86.15 / 97.60
24.71 / 46.56
72.72 / 86.77
79.30 / 92.50
65.72 / 80.86
Restart every 10
87.75 / 95.80
31.71 / 47.33
75.18 / 84.92
81.52 / 94.50
69.04 / 80.64
No restart
88.47 / 95.40
36.04 / 47.33
76.17 / 83.33
82.41 / 93.25
70.77 / 79.83
Appendix
Table 23: Component ablations during exploration (%). Each cell lists the mean accuracy over the 24 steps and the cumulative pass rate at step 24. Overall is the unweighted mean over the four benchmarks. Single run with run seed 1.
Model
Treatment
MATH-500
LCB
MBPP+
SciKE
Overall
Cost
Qwen3-8B
No compression
86.87 / 96.80
28.88 / 46.56
–
–
–
–
Tail (LEAF)
87.00 / 97.00
29.29 / 47.33
74.94 / 85.71
80.88 / 92.75
68.03 / 80.70
1.53 / 1,045
Head and tail
87.09 / 96.40
30.22 / 48.09
74.26 / 86.51
80.76 / 93.25
68.08 / 81.06
1.45 / 1,043
Olmo-3-7B-Instruct
LLM summary
–
31.04 / 56.49
–
75.79 / 98.00
–
–
Tail (LEAF)
93.11 / 97.40
32.63 / 58.78
76.17 / 87.04
75.82 / 97.75
69.43 / 85.24
6.26 / 1,235
Head and tail
93.52 / 97.80
32.35 / 56.49
75.29 / 86.77
76.66 / 98.25
69.45 / 84.83
6.39 / 1,247
Appendix
Table 24: History compression ablation during exploration (%). Each benchmark cell lists the mean accuracy over the 24 steps and the cumulative pass rate at step 24. Cost lists total GPU hours and PFLOPs over the four benchmarks. Dashes mark benchmarks that were not run. Single run with run seed 1.
Model
Update interval
MATH-500
LCB
MBPP+
SciKE
Overall / Total
Mean accuracy / cumulative pass rate (%)
Qwen3-8B
No update
87.12 / 97.20
29.48 / 47.33
74.17 / 86.24
80.52 / 93.00
67.82 / 80.94
Every step
87.49 / 96.60
28.98 / 45.04
74.61 / 86.77
80.97 / 93.50
68.01 / 80.48
Every 2 steps
87.28 / 96.60
29.99 / 46.56
74.86 / 87.04
80.77 / 93.00
68.23 / 80.80
Every 5 steps
87.07 / 96.60
28.31 / 45.04
74.64 / 86.24
80.44 / 94.25
67.61 / 80.53
Olmo-3-7B-Instruct
No update
93.68 / 97.40
32.38 / 55.73
76.17 / 87.57
76.13 / 97.25
69.59 / 84.49
Appendix
Table 25: Interleaved policy updates during exploration, by benchmark. The upper block lists the mean accuracy over the steps and the cumulative pass rate at the last step. The lower block lists the cumulative cost at the last step. The last column gives the Overall value for accuracy and the total over the four benchmarks for cost.
Variant
Mean acc.
Final pass
GPU h
PFLOPs
LEAF
68.705
82.715
3.885
1,159.0
w/o last response
63.395
81.095
3.620
926.0
w/o feedback
67.065
80.780
4.375
1,310.5
Restart every 2
67.235
83.105
4.930
1,382.5
Restart every 10
69.605
82.790
3.640
1,086.5
No restart
70.600
81.815
3.165
1,008.5
Appendix
Table 26: Component ablations averaged over Qwen3-8B and Olmo-3-7B-Instruct, underlying Figure 4 . Mean acc. and Final pass are the averages of the Overall values in Table 23 (%). GPU h and PFLOPs are the averages of the totals in Table 22 .
Model
Training set
Problems
Succ./problem
Fail./problem
Plain prompt
Random
From other problems
Restart
Qwen3-8B
LCB
307
12.93
6.96
69
3,393
1,118
578
MATH
432
15.96
4.02
217
5,567
3,574
1,326
SciKnowEval
334
11.13
3.78
165
2,671
1,698
1,046
Olmo-3-7B-Instruct
LCB
334
11.59
8.40
33
3,223
558
648
MATH
469
18.32
1.63
328
6,932
5,307
1,658
SciKnowEval
375
11.68
8.13
46
3,467
872
912
Appendix
Table 27: Trace statistics on the training problems. Problems is the number of problems with at least one successful context. Succ./problem and Fail./problem are the mean numbers of successful and failure contexts per problem. Plain prompt is the number of problems without a failure context. Random is the number of random-trace contexts with replaced feedback, and From other problems counts those whose feedback comes from another problem. Restart is the number of restart-step contexts kept unchanged.
Qwen3-8B
Olmo-3-7B-Instruct
Set
Model
W → R
R → W
Repeat
Restart
Mean
W → R
R → W
Repeat
Restart
Mean
LCB
Base
26.26
3.59
7.71
45.60
65.80
29.73
16.15
0.17
48.20
57.81
LEAF
14.35
2.72
8.43
57.49
65.60
30.27
12.87
0.10
55.54
63.77
SFT
19.17
10.71
9.00
47.72
55.60
25.26
13.19
0.44
49.55
58.20
MATH
Base
15.92
2.37
64.80
76.50
79.86
24.92
1.10
47.54
89.66
92.24
LEAF
25.84
1.91
43.82
91.44
92.29
26.04
1.15
39.93
89.98
92.60
Appendix
Table 28: Self-correction on the training problems (%). LEAF is the policy trained with successful traces, and SFT is the SFT baseline. W → R is wrong-to-right, R → W is right-to-wrong, and Repeat is the repeated-error rate, all conditioned on the previous step. The wrong-to-wrong rate equals 100− W → R. Restart is the accuracy at restart steps, and Mean is the accuracy over all 10 steps. Single run with run seed 1.
Model
Traces
LCB
MATH
SciKE
Qwen3-8B
Successful
82.98
90.86
83.23
Failure
39.82
60.30
67.07
excl. plain prompt
24.47
26.98
39.64
Random
69.30
88.54
80.16
Olmo-3-7B-Inst.
Successful
71.03
96.16
67.93
Failure
44.76
79.48
50.13
Appendix
Table 29: Teacher accuracy (%) under each trace type, extending Table 5 . The excl. plain prompt rows leave out problems without a failure context.
Model
Divergence
LCB v6
MATH-500
SciKE
Overall
Qwen3-8B
Base
19.08 / 31.30
84.45 / 93.60
74.69 / 80.50
59.41
Reverse KL (LEAF)
26.15 / 37.40
91.75 / 96.60
76.88 / 82.25
64.92
Forward KL
23.66 / 37.40
91.75 / 95.80
76.00 / 85.25
63.80
JSD
24.81 / 33.59
78.65 / 89.20
75.62 / 83.50
59.69
Olmo-3-7B-Instruct
Base
21.56 / 30.53
89.15 / 96.00
68.81 / 80.75
59.84
Reverse KL (LEAF)
24.05 / 33.59
90.40 / 96.00
73.69 / 86.50
62.71
Appendix
Table 30: Divergence ablation by test set (%). Each test cell lists avg@4 and pass@4. Overall is the mean avg@4 over LCB v6, MATH-500 and SciKnowEval, as in Table 6 . Single run with run seed 1.
Model
Div.
Trunc. (L / M / S)
GPU h (L / M / S)
GPU h
EFLOPs
Qwen3-8B
Base
0.00 / 0.05 / 0.00
–
–
–
RKL
0.00 / 6.70 / 0.06
0.47 / 0.95 / 0.62
2.04
1.28
FKL
0.00 / 5.00 / 0.31
0.51 / 0.94 / 0.69
2.15
1.26
JSD
0.00 / 0.00 / 0.31
0.53 / 0.67 / 0.71
1.90
1.12
Olmo-3-7B-I.
Base
2.67 / 3.25 / 0.19
–
–
–
RKL
3.24 / 3.85 / 0.25
1.85 / 1.48 / 0.71
4.04
1.44
Appendix
Table 31: Truncation rate of test responses (%) and training cost of the divergence ablation. L / M / S denote LCB v6 / MATH-500 / SciKnowEval for truncation and the corresponding training sets for GPU hours. The mean of the three truncation rates gives the Trunc. column of Table 6 . Training cost excludes the data rollout.
Model
Set
#Prob.
Base
Update
Rollout
Rel. gain
Qwen3-8B
LCB
307
42.35
58.31
72.64
52.69
MATH
432
73.15
86.34
81.71
154.05
SciKE
334
74.63
82.63
82.04
108.08
Olmo-3-7B-I.
LCB
334
34.13
38.62
60.48
17.05
MATH
469
80.44
83.58
93.82
23.51
SciKE
375
44.47
70.67
61.07
157.83
Appendix
Table 32: Internalization check on the training problems used in the policy update (%). Base, Update and Rollout are the base model, policy update and with-context rollout accuracies. Rel. gain is defined in the text.
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@k scaling, indicating that ExpDis produces models that generate more diverse correct solutions.
Saif Punjwani, Micah Goldblum
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy from the resulting rewards. The policy therefore receives direct learning signals only from trajectories that appear in its rollouts. Yet repeated sampling from the same policy can concentrate on similar reasoning paths, leaving useful alternatives unexplored under a fixed rollout budget. We introduce Activation-Space Exploration (ASE), an online method for diversifying training rollouts. By clustering activation differences between correct and incorrect responses to the same problem, ASE builds a structured bank of latent exploration vectors. Different vectors are then applied to subsequent rollouts to explore alternative reasoning paths. On Qwen3-1.7B-Base and Qwen3-8B-Base trained on DAPO-Math-17k, ASE improves average Mean@8 across six mathematical reasoning benchmarks over GRPO by 4.87 and 1.87 percentage points, respectively, under the same rollout budget.
Yunan Zeng, Jinhao Jing, Xi Cheng +1
Fudan University · The Chinese University of Hong Kong, Shenzhen · Kunlun Inc. +1