Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning

Yutong Chen; Zhichao Xu; Rui Liu; Kunpeng Liu

cs.LGOct 13, 2026

Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning

Authors: Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu

Organizations: Department of Computer Science, Clemson University · Kahlert School of Computing, University of Utah

Abstract

Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: https://github.com/on1262/LEAF

Figures & tables

	Random sampling	SDPO	SCoRe	RLVR	Ours
Rich feedback	✗	✓	✗	✗	✓
Inference adaptation	✗	✗	✓	✗	✓
Single rollout	✓	✗	✗	✗	✓

Figure 1: Properties of exploration methods as evaluated here. Rich feedback : uses textual feedback such as test output, not only the scalar reward. Inference adaptation : the rollout context changes with earlier attempts on the same problem, without a parameter update. Single rollout : one verified answer per problem per step.

Figure 2: Overview of LEAF. Rollout : each problem keeps a memory of its last response, reward and feedback, which forms the context of the next attempt. The memory is updated after each trial. Policy update : after all rollout steps are finished, the teacher conditioned on a successful context teaches the student without context through self-distillation.

Figure 3: Cumulative pass rate on four benchmarks against (a) verifier calls per problem, (b) cumulative compute in PFLOPs (log scale) and (c) cumulative GPU time. Both axes are averaged over the two models at each step.

	Qwen3-8B			Olmo-3-7B-Inst.
Method	Pass	GPU h	PFLOPs	Pass	GPU h	PFLOPs
Random Sampling	77.44	0.66	625	78.15	3.42	676
GRPO	77.78	4.43	2,494	78.00	9.58	2,885
SDPO	77.71	5.06	3,277	78.31	10.46	3,843
SCoRe	78.65	5.09	3,058	82.73	12.33	4,401
SEED	77.96	7.87	4,580	77.47	9.43	3,796

Table 1: Pass rate (%) averaged over the four benchmarks and summed exploration cost. Refer to Appendix D for results by seed and benchmark.

	Qwen3-8B						Olmo-3-7B-Instruct
Method	LCB $\uparrow$	MATH $\uparrow$	SciKE $\uparrow$	Overall $\uparrow$	GPU h $\downarrow$	EFLOPs $\downarrow$	LCB $\uparrow$	MATH $\uparrow$	SciKE $\uparrow$	Overall $\uparrow$	GPU h $\downarrow$	EFLOPs $\downarrow$
Base	19.78	84.08	74.40	59.42	–	–	21.25	89.12	69.25	59.87	–	–
GRPO	26.02	86.53	76.75	63.10	5.52	3.71	21.12	89.33	70.83	60.43	14.06	4.52
SDPO	25.64	81.65	72.96	60.08	5.82	4.35	23.66	82.38	66.98	57.68	14.23	5.93
SCoRe	22.71	84.32	75.17	60.73	5.57	4.17	20.48	88.95	68.38	59.27	17.71	6.33
SEED	25.89	85.58	76.83	62.77	6.72	4.88	22.33	89.10	72.10	61.18	14.56	5.94

Table 2: In-distribution generalization results. Avg@4 (%) on test sets. LCB: LiveCodeBench v6. MATH: MATH-500. SciKE: SciKnowEval. GPU h and EFLOPs (

10^{18}

FLOPs) sum the three training runs. For LEAF and SFT they include the exploration rollout that collects the training data, and for SFT also rewriting and verifying the solutions. Refer to Appendix F for results by run seed and training cost by stage.

Figure 4: Exploration ablations on key components. Final pass rate against exploration GPU time, averaged over the two models. Our method in main experiments is denoted as red star.

	(a) Exploration		(b) Generalization
Method	Length	Gain (pp.)	Length	Gain (pp.)
GRPO	1.00	0.09	1.14	2.12
SDPO	0.99	0.21	0.98	$-$ 0.77
SCoRe	1.27	2.89	1.01	0.35
SEED	1.00	0.31	1.11	2.33
SFT	–	–	0.93	$-$ 1.46

Table 3: Response length and accuracy gain over the reference. Length: ratio of the mean response length to that of the reference. Gain: accuracy gain over the reference in pp.

	Qwen3-8B		Olmo-3-7B-Inst.
Method	Pass	GPU h	Pass	GPU h
Random Sampling (ref.)	77.06	0.64	77.81	3.39
Ours (no update)	80.94	1.47	84.49	6.30
Ours + update every step	80.48	5.55	85.16	13.97
Ours + update every 2 steps	80.80	4.38	84.14	11.40
Ours + update every 5 steps	80.53	3.40	84.25	9.70

Table 4: Final cumulative pass rate (%, averaged over the four benchmarks) and exploration GPU hours.

	Qwen3-8B			Olmo-3-7B-Inst.
Traces	LCB	MATH	SciKE	LCB	MATH	SciKE
Successful	82.98	90.86	83.23	71.03	96.16	67.93
Failure	39.82	60.30	67.07	44.76	79.48	50.13
Random	69.30	88.54	80.16	69.24	93.76	66.80

Table 5: Teacher accuracy (%) on the training problems under different rollout traces.

	Qwen3-8B		Olmo-3-7B-Inst.
Method	Overall	Trunc.	Overall	Trunc.
Base	59.41	0.02	59.84	2.04
Ours (reverse KL)	64.92	2.25	62.71	2.45
Ours w/ forward KL	63.80	1.77	60.93	2.02
Ours w/ JSD	59.69	0.10	62.04	1.83

Table 6: Ablations on three loss types. Overall test avg@4 (%) and truncated test responses (%, mean over the three test sets).

	Qwen3-8B		Olmo-3-7B-Instruct
Benchmark	Wrong	Repeat (%)	Wrong	Repeat (%)
MATH-500	1,912	45.7	1,309	47.4
LCB	2,480	61.8	2,452	40.5
MBPP+	1,278	77.3	1,675	78.0

Table 7: Repeated wrong answers among the 24 independent Random Sampling responses per problem. Wrong is the total number of wrong responses over all problems. Repeat is the fraction of wrong responses whose error signature equals that of an earlier wrong response to the same problem. Correctness on LCB and MBPP+ is judged on public tests.

Model	Benchmark	Comparison	Problems	Repeat (%)	$\Delta$ Repeat (pp)	Fix (%)	$\Delta$ Fix (pp)
Qwen3-8B	MATH-500	w/o feedback	143	52.5 / 65.0	$-12.5$ ( $-17.9$ , $-7.4$ )	29.5 / 19.8	$+9.7$ ( $+4.2$ , $+15.1$ )
	MATH-500	w/o last response	148	51.6 / 12.4	$+39.2$ ( $+34.1$ , $+44.4$ )	30.9 / 53.0	$-22.2$ ( $-28.0$ , $-16.7$ )
	LCB	w/o feedback	115	38.3 / 67.6	$-29.4$ ( $-35.8$ , $-23.3$ )	16.6 / 3.5	$+13.0$ ( $+8.2$ , $+18.1$ )
	LCB	w/o last response	117	38.2 / 23.8	$+14.4$ ( $+9.6$ , $+18.9$ )	17.3 / 16.0	$+1.3$ ( $-1.7$ , $+4.4$ )
	MBPP+	w/o feedback	114	36.5 / 79.2	$-42.7$ ( $-49.0$ , $-36.0$ )	54.6 / 16.7	$+37.9$ ( $+30.9$ , $+44.9$ )
	MBPP+	w/o last response	126	36.0 / 27.3	$+8.7$ ( $+2.8$ , $+15.1$ )	55.4 / 60.5	$-5.1$ ( $-11.9$ , $+1.2$ )

Table 8: Repeat and fix rates of LEAF compared with two memory ablations over the first 24 exploration steps. Repeat and Fix entries are listed as LEAF / comparison, averaged per problem over transitions and then over the problems shared by both conditions (Problems).

\Delta

is LEAF minus the comparison in percentage points, with the 95% bootstrap confidence interval over problems in parentheses.

		LEAF			LEAF w/o feedback
Model	Benchmark	Repeat (%)	Base (%)	Ratio	Repeat (%)	Base (%)	Ratio
Qwen3-8B	MATH-500	60.9	16.6	3.67	73.4	17.3	4.24
	LCB	41.8	23.3	1.79	66.8	30.2	2.21
	MBPP+	55.4	40.9	1.35	85.0	38.1	2.23
Olmo-3-7B-Instruct	MATH-500	56.0	21.0	2.67	65.0	23.8	2.73
	LCB	27.4	9.7	2.84	47.1	10.5	4.47

Table 9: Repeat rates relative to independent sampling, computed over all transitions of each condition together. Base is the fraction of the 24 Random Sampling responses to the same problem that are wrong with the same error signature as the previous answer, averaged over transitions. Ratio is the number of repeats divided by the sum of base frequencies.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Random Sampling	GRPO	SDPO	SCoRe	SEED	LEAF
Verifier calls / problem	24	24	24	24	24	24
GPU-hour cap / benchmark	5	5	5	5	5	5
Parameter updates	✗	✓	✓	✓	✓	✗
Questions / update	–	32	32	32	32	–
Responses / question	1 per step	8	8	$4\times 2$ turns	8	1 verified per step (+1 draft at restart)
Updates	–	47/13/36/38	47/13/36/38	47/13/36/38	8/2/6/6 (collect) + 39/11/30/32 (RL)	–

Appendix

Table 10: Exploration hyperparameters for all methods. The response limit is 16,384 tokens for Olmo-3-7B-Instruct on LCB, and SCoRe limits the two turns to 16,896 tokens in total. All methods with parameter updates use AdamW with weight decay 0.01, a maximum gradient norm of 1.0, and a constant learning rate. Dashes mark settings that do not apply or are not set separately for that method.

		MATH-500		LCB		MBPP+		SciKnowEval		Overall
Model	Method	Acc.	Pass	Acc.	Pass	Acc.	Pass	Acc.	Pass	Acc.	Pass
Qwen3-8B	Random Sampling	84.14	97.33	20.14	41.48	72.71	85.80	74.83	85.17	62.96	77.44
	GRPO	84.01	96.87	19.99	43.00	73.15	86.07	74.83	85.17	63.00	77.78
	SDPO	83.97	97.27	19.77	42.75	72.98	86.07	74.76	84.75	62.87	77.71
	SCoRe	84.45	96.13	20.42	41.98	72.16	85.89	76.35	90.58	63.34	78.65
	SEED	84.04	97.33	19.72	43.77	72.96	85.89	74.85	84.83	62.89	77.96

Appendix

Table 11: Exploration results by benchmark, averaged over three run seeds (%). Acc. is the average accuracy over all answers up to the cutoff step, and Pass is the cumulative pass rate at that step. Overall is the macro average over the four benchmarks. The cutoff step of a run is the last step reached by all four benchmarks, as in Table 1 . It is step 16 for SEED on Olmo-3-7B-Instruct at seeds 1 and 2 and step 24 otherwise.

		Pass (%) by seed			Mean over 3 seeds
Model	Method	Seed 1	Seed 2	Seed 3	Pass (%)	GPU h	PFLOPs
Qwen3-8B	Random Sampling	77.06	77.53	77.74	$77.44\pm 0.34$	$0.66\pm 0.03$	$625\pm 1$
	GRPO	78.18	76.97	78.18	$77.78\pm 0.70$	$4.43\pm 0.09$	$2{,}494\pm 3$
	SDPO	78.35	76.98	77.79	$77.71\pm 0.69$	$5.06\pm 0.10$	$3{,}277\pm 5$
	SCoRe	78.52	78.85	78.57	$78.65\pm 0.18$	$5.09\pm 0.06$	$3{,}058\pm 5$
	SEED	77.87	77.73	78.26	$77.96\pm 0.27$	$7.87\pm 0.16$	$4{,}580\pm 8$

Appendix

Table 12: Exploration pass rate (%) by run seed, macro-averaged over MATH-500, LCB, MBPP+, and SciKnowEval, with the summed exploration cost underlying Table 1 . The last three columns are means

\pm

sample standard deviations over 3 seeds. † The MATH-500 cell stopped at 16 steps, so the values are taken at step 16.

Model	Method	MATH-500	LCB	MBPP+	SciKnowEval	Total
Qwen3-8B	Random Sampling	0.30 / 199 / 12.1	0.09 / 73 / 4.4	0.10 / 88 / 5.4	0.17 / 265 / 16.2	0.66 / 625 / 38.1
	GRPO	1.82 / 998 / 12.2	1.01 / 362 / 4.4	0.67 / 441 / 5.4	0.92 / 693 / 7.3	4.43 / 2,494 / 29.3
	SDPO	2.04 / 1,351 / 12.0	1.10 / 477 / 4.4	0.84 / 584 / 5.4	1.08 / 865 / 7.4	5.06 / 3,277 / 29.2
	SCoRe	2.07 / 1,152 / 10.8	1.03 / 413 / 3.8	0.85 / 570 / 5.3	1.14 / 923 / 6.6	5.09 / 3,058 / 26.4
	SEED	3.20 / 1,889 / 15.0	1.28 / 665 / 5.4	1.53 / 869 / 6.7	1.88 / 1,157 / 8.7	7.87 / 4,580 / 35.7
	LEAF	0.48 / 350 / 21.4	0.46 / 140 / 8.5	0.22 / 172 / 10.5	0.31 / 381 / 23.3	1.47 / 1,043 / 63.7

Appendix

Table 13: Exploration cost by benchmark, averaged over three run seeds. Each cell lists GPU h / PFLOPs / generated tokens (M), accumulated up to the cutoff step defined in Table 11 . Total is the sum over the four benchmarks.

$x$ -axis	Seed 1	Seed 2	Seed 3	Mean	Mean curve
Verifier calls	1.97	2.20	1.63	$1.94\pm 0.29$	1.90
PFLOPs	6.19	6.91	5.25	$6.12\pm 0.83$	5.99
GPU time	3.85	4.10	3.33	$3.76\pm 0.40$	3.70

Appendix

Table 14: Speedup of LEAF over SCoRe (

\times

) on the two-model average curves of Figure 3 . We report the mean

\pm

standard deviation of the per-seed ratios over 3 seeds. Mean curve is the ratio on the curve averaged over the three seeds.

		Qwen3-8B		Olmo-3-7B-Instruct
Training set	Problems	$\|\{x:\mathcal{C}^{+}(x)\neq\emptyset\}\|$	$(c^{+},y^{+})$ pairs	$\|\{x:\mathcal{C}^{+}(x)\neq\emptyset\}\|$	$(c^{+},y^{+})$ pairs
LCB (v2–v5)	478	303.0	3,965	335.3	3,921
MATH (Level 5)	500	432.0	6,895	467.3	8,567
SciKnowEval (L5)	400	335.3	3,694	377.3	4,407

Appendix

Table 15: Training sets for in-distribution generalization and the data collected by the

K=20

step exploration rollout of LEAF, averaged over three run seeds.

|\{x:\mathcal{C}^{+}(x)\neq\emptyset\}|

is the number of problems with at least one successful context, which is also the number of problems used in the policy update.

(c^{+},y^{+})

pairs counts the distinct correct, untruncated answers with their contexts, summed over problems.

Model	Training set	Pairs	Kept	Dropped	Kept per round (1/2/3/4/5)	Length	Trunc. (%)	GPU h
Qwen3-8B	LCB	3,965	3,959	6	3,940 / 18 / 1 / 0 / 0	349	0.00	0.44
	MATH	6,895	6,895	0	6,889 / 5 / 1 / 0 / 0	737	0.01	0.37
	SciKnowEval	3,694	3,588	106	3,415 / 96 / 41 / 22 / 15	1,031	0.03	0.39
Olmo-3-7B-Instruct	LCB	3,921	3,899	22	3,823 / 57 / 12 / 5 / 2	404	0.00	0.58
	MATH	8,567	8,566	1	8,562 / 2 / 0 / 1 / 0	597	0.00	0.95
	SciKnowEval	4,407	3,964	443	3,031 / 484 / 227 / 136 / 86	899	3.96	1.49

Appendix

Table 16: Statistics of the rewritten solutions

y^{*}

used by the SFT baseline, averaged over three run seeds and rounded. Pairs is the number of

(c^{+},y^{+})

pairs. Kept and Dropped count pairs whose rewrite did or did not pass the verifier within 5 rounds. Kept per round gives the number of pairs first accepted in each round. Length is the mean number of tokens of the kept rewrites. Trunc. is the fraction of truncated rewrite generations over all rounds. GPU h covers rewriting and verification.

Model	Method	LCB	MATH	SciKE	Overall
Qwen3-8B	Base	19.08 / 18.89 / 21.37	84.45 / 83.60 / 84.20	74.69 / 74.00 / 74.50	59.41 / 58.83 / 60.02
	GRPO	25.38 / 25.95 / 26.72	86.35 / 86.90 / 86.35	75.62 / 77.25 / 77.38	62.45 / 63.37 / 63.48
	SDPO	25.57 / 25.57 / 25.76	82.00 / 81.40 / 81.55	73.56 / 72.12 / 73.19	60.38 / 59.70 / 60.17
	SCoRe	22.52 / 22.52 / 23.09	84.20 / 84.70 / 84.05	75.31 / 75.06 / 75.12	60.68 / 60.76 / 60.76
	SEED	26.34 / 25.00 / 26.34	86.20 / 84.55 / 86.00	76.75 / 77.19 / 76.56	63.10 / 62.25 / 62.97
	SFT	22.52 / 23.47 / 20.80	77.40 / 77.00 / 76.15	76.00 / 75.50 / 74.62	58.64 / 58.66 / 57.19

Appendix

Table 17: Test avg@4 results by run seed (%). Values are ordered by run seed 1 / 2 / 3. Overall is the mean of LCB, MATH and SciKE for the same run seed.

Model	Method	LCB	MATH	SciKE	Overall
Qwen3-8B	Base	19.78 / 31.55	84.08 / 93.53	74.40 / 81.00	59.42
	GRPO	26.02 / 35.62	86.53 / 94.33	76.75 / 82.33	63.10
	SDPO	25.64 / 34.10	81.65 / 91.40	72.96 / 80.58	60.08
	SCoRe	22.71 / 32.82	84.32 / 93.60	75.17 / 81.92	60.73
	SEED	25.89 / 35.88	85.58 / 93.87	76.83 / 82.42	62.77
	SFT	22.26 / 33.08	76.85 / 88.87	75.38 / 84.83	58.16

Appendix

Table 18: Test avg@4 / pass@4 (%), averaged over three run seeds. Overall is the mean avg@4 over LCB, MATH and SciKE.

		GPU h							EFLOPs
Model	Method	LCB	MATH	SciKE	Train	Rollout	Rewrite	Total	Train	Rollout	Rewrite	Total
Qwen3-8B	GRPO	1.78	2.42	1.31	5.52	–	–	5.52	3.71	–	–	3.71
	SDPO	1.84	2.03	1.96	5.82	–	–	5.82	4.35	–	–	4.35
	SCoRe	1.69	2.40	1.47	5.57	–	–	5.57	4.17	–	–	4.17
	SEED	2.21	2.79	1.71	6.72	–	–	6.72	4.88	–	–	4.88
	SFT	0.16	0.17	0.22	0.55	1.58	1.19	3.32	0.39	1.36	0.69	2.44

Appendix

Table 19: Training cost by training set and stage, averaged over three run seeds. LCB, MATH and SciKE give the training GPU h on each training set, and Train is their sum. Rollout is the exploration rollout that collects the training data of LEAF and SFT, and Rewrite is the generation and verification of the SFT targets. Total GPU h and Total EFLOPs (

10^{18}

FLOPs) equal the values in Table 2 . – marks a stage that the method does not use.

	Qwen3-8B				Olmo-3-7B-Instruct
Method	MATH-500	LCB	MBPP+	SciKE	MATH-500	LCB	MBPP+	SciKE
Random Sampling	883	841	395	344	1,175	2,821	483	207
GRPO	886	839	395	345	1,175	2,792	478	206
SDPO	871	835	391	353	1,171	2,767	471	204
SCoRe	802	900	455	395	1,310	3,266	853	385
SEED	885	836	392	346	1,159	2,829	485	209

Appendix

Table 20: Mean response tokens per graded answer during exploration, averaged over three run seeds.

	Qwen3-8B			Olmo-3-7B-Instruct
Method	LCB	MATH-500	SciKE	LCB	MATH-500	SciKE
Base	843 / 0.00	879 / 0.12	341 / 0.00	2,781 / 2.61	1,166 / 2.95	205 / 0.15
GRPO	936 / 0.00	1,001 / 0.35	412 / 0.00	3,049 / 2.54	1,201 / 3.10	254 / 0.29
SDPO	808 / 0.00	678 / 0.00	471 / 1.40	3,674 / 2.42	747 / 1.00	171 / 0.02
SCoRe	849 / 0.00	878 / 0.10	350 / 0.04	2,765 / 1.72	1,190 / 3.23	205 / 0.13
SEED	828 / 0.06	925 / 0.12	394 / 0.02	2,677 / 1.27	1,219 / 3.20	304 / 0.90

Appendix

Table 21: Test response length after the policy update. Each cell lists the mean completion tokens and the truncation rate (%), both averaged over three run seeds. Means include truncated responses.

Model	Variant	MATH-500	LCB	MBPP+	SciKE	Total
Qwen3-8B	LEAF	0.48 / 350 / 21.4	0.46 / 140 / 8.6	0.22 / 173 / 10.6	0.32 / 380 / 23.2	1.47 / 1,044 / 63.7
	w/o last response	0.55 / 280 / 17.1	0.47 / 108 / 6.6	0.17 / 139 / 8.5	0.27 / 326 / 19.9	1.47 / 854 / 52.1
	w/o feedback	0.52 / 371 / 22.6	0.49 / 140 / 8.5	0.20 / 194 / 11.8	0.35 / 434 / 26.5	1.56 / 1,139 / 69.5
	Restart every 2	0.63 / 421 / 25.7	0.46 / 164 / 10.0	0.25 / 213 / 13.0	0.33 / 419 / 25.6	1.68 / 1,217 / 74.3
	Restart every 10	0.43 / 331 / 20.2	0.50 / 134 / 8.2	0.21 / 160 / 9.8	0.29 / 367 / 22.4	1.43 / 992 / 60.6
	No restart	0.39 / 314 / 19.2	0.52 / 124 / 7.5	0.20 / 145 / 8.9	0.25 / 351 / 21.4	1.36 / 934 / 57.0

Appendix

Table 22: Exploration cost of the component ablations. Each cell lists GPU hours, PFLOPs and generated tokens (millions), cumulative at the last step of the benchmark. Total is the sum over the four benchmarks.

Model	Variant	MATH-500	LCB	MBPP+	SciKE	Overall
Qwen3-8B	LEAF	87.12 / 97.20	29.48 / 47.33	74.17 / 86.24	80.52 / 93.00	67.82 / 80.94
	w/o last response	84.07 / 98.00	21.56 / 46.56	71.88 / 86.24	77.02 / 88.00	63.63 / 79.70
	w/o feedback	85.92 / 95.60	22.30 / 38.93	71.10 / 83.60	80.67 / 93.25	65.00 / 77.84
	Restart every 2	86.15 / 97.60	24.71 / 46.56	72.72 / 86.77	79.30 / 92.50	65.72 / 80.86
	Restart every 10	87.75 / 95.80	31.71 / 47.33	75.18 / 84.92	81.52 / 94.50	69.04 / 80.64
	No restart	88.47 / 95.40	36.04 / 47.33	76.17 / 83.33	82.41 / 93.25	70.77 / 79.83

Appendix

Table 23: Component ablations during exploration (%). Each cell lists the mean accuracy over the 24 steps and the cumulative pass rate at step 24. Overall is the unweighted mean over the four benchmarks. Single run with run seed 1.

Model	Treatment	MATH-500	LCB	MBPP+	SciKE	Overall	Cost
Qwen3-8B	No compression	86.87 / 96.80	28.88 / 46.56	–	–	–	–
	Tail (LEAF)	87.00 / 97.00	29.29 / 47.33	74.94 / 85.71	80.88 / 92.75	68.03 / 80.70	1.53 / 1,045
	Head and tail	87.09 / 96.40	30.22 / 48.09	74.26 / 86.51	80.76 / 93.25	68.08 / 81.06	1.45 / 1,043
Olmo-3-7B-Instruct	LLM summary	–	31.04 / 56.49	–	75.79 / 98.00	–	–
	Tail (LEAF)	93.11 / 97.40	32.63 / 58.78	76.17 / 87.04	75.82 / 97.75	69.43 / 85.24	6.26 / 1,235
	Head and tail	93.52 / 97.80	32.35 / 56.49	75.29 / 86.77	76.66 / 98.25	69.45 / 84.83	6.39 / 1,247

Appendix

Table 24: History compression ablation during exploration (%). Each benchmark cell lists the mean accuracy over the 24 steps and the cumulative pass rate at step 24. Cost lists total GPU hours and PFLOPs over the four benchmarks. Dashes mark benchmarks that were not run. Single run with run seed 1.

Model	Update interval	MATH-500	LCB	MBPP+	SciKE	Overall / Total
Mean accuracy / cumulative pass rate (%)
Qwen3-8B	No update	87.12 / 97.20	29.48 / 47.33	74.17 / 86.24	80.52 / 93.00	67.82 / 80.94
	Every step	87.49 / 96.60	28.98 / 45.04	74.61 / 86.77	80.97 / 93.50	68.01 / 80.48
	Every 2 steps	87.28 / 96.60	29.99 / 46.56	74.86 / 87.04	80.77 / 93.00	68.23 / 80.80
	Every 5 steps	87.07 / 96.60	28.31 / 45.04	74.64 / 86.24	80.44 / 94.25	67.61 / 80.53
Olmo-3-7B-Instruct	No update	93.68 / 97.40	32.38 / 55.73	76.17 / 87.57	76.13 / 97.25	69.59 / 84.49

Appendix

Table 25: Interleaved policy updates during exploration, by benchmark. The upper block lists the mean accuracy over the steps and the cumulative pass rate at the last step. The lower block lists the cumulative cost at the last step. The last column gives the Overall value for accuracy and the total over the four benchmarks for cost.

Variant	Mean acc.	Final pass	GPU h	PFLOPs
LEAF	68.705	82.715	3.885	1,159.0
w/o last response	63.395	81.095	3.620	926.0
w/o feedback	67.065	80.780	4.375	1,310.5
Restart every 2	67.235	83.105	4.930	1,382.5
Restart every 10	69.605	82.790	3.640	1,086.5
No restart	70.600	81.815	3.165	1,008.5

Appendix

Table 26: Component ablations averaged over Qwen3-8B and Olmo-3-7B-Instruct, underlying Figure 4 . Mean acc. and Final pass are the averages of the Overall values in Table 23 (%). GPU h and PFLOPs are the averages of the totals in Table 22 .

Model	Training set	Problems	Succ./problem	Fail./problem	Plain prompt	Random	From other problems	Restart
Qwen3-8B	LCB	307	12.93	6.96	69	3,393	1,118	578
	MATH	432	15.96	4.02	217	5,567	3,574	1,326
	SciKnowEval	334	11.13	3.78	165	2,671	1,698	1,046
Olmo-3-7B-Instruct	LCB	334	11.59	8.40	33	3,223	558	648
	MATH	469	18.32	1.63	328	6,932	5,307	1,658
	SciKnowEval	375	11.68	8.13	46	3,467	872	912

Appendix

Table 27: Trace statistics on the training problems. Problems is the number of problems with at least one successful context. Succ./problem and Fail./problem are the mean numbers of successful and failure contexts per problem. Plain prompt is the number of problems without a failure context. Random is the number of random-trace contexts with replaced feedback, and From other problems counts those whose feedback comes from another problem. Restart is the number of restart-step contexts kept unchanged.

		Qwen3-8B					Olmo-3-7B-Instruct
Set	Model	W $\to$ R	R $\to$ W	Repeat	Restart	Mean	W $\to$ R	R $\to$ W	Repeat	Restart	Mean
LCB	Base	26.26	3.59	7.71	45.60	65.80	29.73	16.15	0.17	48.20	57.81
	LEAF	14.35	2.72	8.43	57.49	65.60	30.27	12.87	0.10	55.54	63.77
	SFT	19.17	10.71	9.00	47.72	55.60	25.26	13.19	0.44	49.55	58.20
MATH	Base	15.92	2.37	64.80	76.50	79.86	24.92	1.10	47.54	89.66	92.24
	LEAF	25.84	1.91	43.82	91.44	92.29	26.04	1.15	39.93	89.98	92.60

Appendix

Table 28: Self-correction on the training problems (%). LEAF is the policy trained with successful traces, and SFT is the SFT baseline. W

\to

R is wrong-to-right, R

\to

W is right-to-wrong, and Repeat is the repeated-error rate, all conditioned on the previous step. The wrong-to-wrong rate equals

100-

\to

R. Restart is the accuracy at restart steps, and Mean is the accuracy over all 10 steps. Single run with run seed 1.

Model	Traces	LCB	MATH	SciKE
Qwen3-8B	Successful	82.98	90.86	83.23
	Failure	39.82	60.30	67.07
	excl. plain prompt	24.47	26.98	39.64
	Random	69.30	88.54	80.16
Olmo-3-7B-Inst.	Successful	71.03	96.16	67.93
	Failure	44.76	79.48	50.13

Appendix

Table 29: Teacher accuracy (%) under each trace type, extending Table 5 . The excl. plain prompt rows leave out problems without a failure context.

Model	Divergence	LCB v6	MATH-500	SciKE	Overall
Qwen3-8B	Base	19.08 / 31.30	84.45 / 93.60	74.69 / 80.50	59.41
	Reverse KL (LEAF)	26.15 / 37.40	91.75 / 96.60	76.88 / 82.25	64.92
	Forward KL	23.66 / 37.40	91.75 / 95.80	76.00 / 85.25	63.80
	JSD	24.81 / 33.59	78.65 / 89.20	75.62 / 83.50	59.69
Olmo-3-7B-Instruct	Base	21.56 / 30.53	89.15 / 96.00	68.81 / 80.75	59.84
	Reverse KL (LEAF)	24.05 / 33.59	90.40 / 96.00	73.69 / 86.50	62.71

Appendix

Table 30: Divergence ablation by test set (%). Each test cell lists avg@4 and pass@4. Overall is the mean avg@4 over LCB v6, MATH-500 and SciKnowEval, as in Table 6 . Single run with run seed 1.

Model	Div.	Trunc. (L / M / S)	GPU h (L / M / S)	GPU h	EFLOPs
Qwen3-8B	Base	0.00 / 0.05 / 0.00	–	–	–
	RKL	0.00 / 6.70 / 0.06	0.47 / 0.95 / 0.62	2.04	1.28
	FKL	0.00 / 5.00 / 0.31	0.51 / 0.94 / 0.69	2.15	1.26
	JSD	0.00 / 0.00 / 0.31	0.53 / 0.67 / 0.71	1.90	1.12
Olmo-3-7B-I.	Base	2.67 / 3.25 / 0.19	–	–	–
	RKL	3.24 / 3.85 / 0.25	1.85 / 1.48 / 0.71	4.04	1.44

Appendix

Table 31: Truncation rate of test responses (%) and training cost of the divergence ablation. L / M / S denote LCB v6 / MATH-500 / SciKnowEval for truncation and the corresponding training sets for GPU hours. The mean of the three truncation rates gives the Trunc. column of Table 6 . Training cost excludes the data rollout.

Model	Set	#Prob.	Base	Update	Rollout	Rel. gain
Qwen3-8B	LCB	307	42.35	58.31	72.64	52.69
	MATH	432	73.15	86.34	81.71	154.05
	SciKE	334	74.63	82.63	82.04	108.08
Olmo-3-7B-I.	LCB	334	34.13	38.62	60.48	17.05
	MATH	469	80.44	83.58	93.82	23.51
	SciKE	375	44.47	70.67	61.07	157.83

Appendix

Table 32: Internalization check on the training problems used in the policy update (%). Base, Update and Rollout are the base model, policy update and with-context rollout accuracies. Rel. gain is defined in the text.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,078 papers

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Jul 30, 2026Shuang Liang, Haoyang Zhou, Yifan Gong +2Numerical Reasoning in Language Models Prompt Selection for RL
Renmin University of China · AntGroup
Decoupling Exploration from Optimization in RLVR
Oct 7, 2026Saif Punjwani, Micah GoldblumRL for Language Model Reasoning RL Exploration
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards
Oct 19, 2026Yunan Zeng, Jinhao Jing, Xi Cheng +1RL for Language Model Reasoning RL Exploration
Fudan University · The Chinese University of Hong Kong, Shenzhen · Kunlun Inc. +1

Jul 30, 2026J/K move · Enter open · S save

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Shuang Liang, Haoyang Zhou, Yifan Gong, Guowei Wang+1

Renmin University of China · AntGroup

Numerical Reasoning in Language Models Prompt Selection for RL Reinforcement Learning with Verifiable Rewards Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.

Abstract

RL for Language Model Reasoning

Explore similar work

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Decoupling Exploration from Optimization in RLVR

Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards